Globe and Mail Survey is BS

Canadian Universities Forum (discussion group)


 
 
Subject: Globe and Mail Survey is BS
"Cynics will say that the Globe and Mail ratings are little more than a expedient venture to capture some of the lucrative market that Maclean?s magazine has created for itself with its annual ranking of Canadian universities. However, the arrival of the echo boomers, the Ontario double cohort, and rising admissions requirements and tuition fees in some provinces are driving the need to consider attending universities people know little about.

Criticism of the Maclean?s rankings has been long running and considerable, with different institutions refusing to participate at various times. On the very day that the Globe and Mail published its ?report card,? Robert Birgeneau, President of the University of Toronto (ranked number one on the Maclean?s scale since 1994) stated that the institution was reconsidering its participation. The most fundamental flaw with Maclean?s is that the great majority of its survey elements measure inputs?how many books the library has, average grade of entering students, operating budgets, etc.?not outputs such as career satisfaction, employability, and success at graduate schools.

Similarly, concern about the Globe and Mail ?report card? ought to centre on its methodology. The Uthink/Strategic Counsel survey published in the Globe and Mail was conducted using the web. Such a survey needs to ensure that respondents are representative of the wider population that the designers are interested in. The Globe and Mail reports that personal validation measures were designed into the survey instrument, but the very fact that only students looking for scholarship information on one web site were primarily targeted means that the sample excludes students who took other approaches to finding out about scholarship information and must have an under representation of students who already hold scholarships or are sponsored (such as First Nations/Aboriginal students). Claims that the survey is representative are likely based on a comparison of the polled population to national demographic statistics, not institutional ones: the results for the University of Saskatchewan are not necessarily representative of the UofS student body.

Campaigning can also undermine the validity of the data collected when a group looking for a certain survey result gets like-minded people to respond. The Uthink/Strategic Counsel respondents were self-selected, the survey was run over a three and half week period, and respondents were encouraged to get friends (likely individuals with similar characteristics) to respond. Hopefully, the survey length (100 items) undermined any efforts at intentional campaigning.

All pollsters need a sample size large enough to extrapolate the findings to the larger population. Therefore, the Globe and Mail student survey designers decided not to include universities from which fewer than 250 students had responded. Some of Canada?s most interesting (and, according to other rankings, best universities) have very low enrolments. A university ranking that excludes these institutions must be of dubious interest.

However, a major difficulty with the Globe and Mail survey is similar to the problem that many voice about the Maclean?s rankings: it simply does not attempt to measure the right things to get the answer it seeks. A survey of current students cannot provide much insight into the output of a university?s educational efforts. Current students simply are not in a good position to comment upon the value of their education because they have not yet completed it or had a meaningful opportunity to put it to use. Diners cannot affirm that they are satisfied with their meal when they are still on the first course.

Most university students attend only one institution and, therefore, do not have external reference points against which to assess the institution at which they study. The parameters for satisfaction assessments will often be internal. In other words, satisfaction surveys of university students can really only produce absolute not relative results: they can only be used to assess how well an institution is satisfying student needs, they cannot be used to compare how well one university meets student needs compared to another university. Getting a university education is not like buying a can of cola where consumers have had multiple opportunities to try Coke, Pepsi, and a variety of other products in that market."

http://www.ccaecanada.org/ensembleMcLeod2.htm

========================================================

Listen, at least with Maclean?s you are given a set of objective statistics that are easily reproducible like entering students average, library holdings, faculty awards, students awards, etc. You can ignore Maclean?s method of combining and weighing these statistics to form a ranking and instead analyze and compare the stats on your own.

But with the Globe and Mail, as you will see, the grades assigned often look completely arbitrary. The sample was not random which is basically the dumbest mistake you can make, and the majority of questions students are asked are completely useless since (1) students have no way of comparing schools (2) students are in no way informed or prepared to answer, etc.

Case and Point:

Library:
-----------------------------------------

York: A-
--------
holdings per student: 131
total holdings: less than a million

UofT: A-
--------
holdings per student: 252
total holdings: 14.873 million

Attractiveness of Campus
------------------------------------------

York: B
--------

York is located next to Jane and Finch - Toronto?s most dangerous area (http://en.wikipedia.org/wiki/Jane_and_Finch).

http://www.yorku.ca/mediar/archive/photos/Ross%20Bldg%20and%20Ramp.jpg
http://www.yorku.ca/mediar/archive/photos/demolition.jpg

UofT: B
--------

UofT is next to Yorkville - One of Toronto?s most affluent neighbourhoods(http://en.wikipedia.org/wiki/Yorkville%2C_Toronto), Bay St - Canada?s "Wall St." (http://en.wikipedia.org/wiki/Bay_Street), and yes, China town.

http://www.flickr.com/photo_zoom.gne?id=173801576&context=set-72157594165309950&size=l
http://www.flickr.com/photos/ming_chai/188469102/
http://www.flickr.com/photo_zoom.gne?id=30014186&size=o
http://www.flickr.com/photo_zoom.gne?id=162993193&size=o
http://www.flickr.com/photos/49459728@N00/172407054/
http://www.flickr.com/photos/54549305@N00/72066672/
http://www.flickr.com/photos/43517501@N00/162851811/
http://www.flickr.com/photos/yoshiboshi/15322074/
http://www.flickr.com/photos/r_tenenbaum/153146834/
http://www.flickr.com/photos/roens/142503019/
http://www.flickr.com/photo_zoom.gne?id=142503235&size=o
http://www.flickr.com/photos/jennyfleur/12318362/
http://www.flickr.com/photos/bifu/182913838/
http://www.flickr.com/photos/mvisosky/19213642/
http://www.flickr.com/photos/mvisosky/19213641/in/photostream/
http://www.flickr.com/photos/viviloob/173520518/in/set-72157594175762069/
http://www.flickr.com/photos/aarthyr/11412160/
http://www.flickr.com/photos/kwisatch/46410037/

I challenge anyone to find better looking architecture than UofT at any Canadian University. UofT is modelled after Yale, and Yale is said to be the best looking campus in the US. Why do you think there are so many pictures of UofT on the net? Why so many tours? So many movies?

Still think UofT deserves a B?

Overall Educational Experience:
----------------------------------------------

York: B+
---------

Large class sizes (same as UofT, UBC, McMaster, Queen?s, and McGill)

Classes taught by tenured faculty: 57.9%
Percentage of Faculty with PhD?s: 90.9%
Awards per faculty (per 1000): 5.5
Entering Grade: 81.0%
Student Awards (per 1000): 3.2


UofT: B+
--------

Classes taught by tenured faculty: 68.3%
Percentage of Faculty with PhD?s: 98.5%
Awards per faculty (per 1000): 10.4
Entering Grade: 85.1%
Student Awards (per 1000): 7.6

-------------------------------------------------

In the end, how do you ask students to rate an entire Universities reputation for cutting edge research? Overall Educational Experience? Reputation of university among employers? Overall quality of career preparation? when the students being polled are still in University and are in no way informed about these things?

Globe and Mail report card is nothing more then an attempt to copy Maclean?s success and make some money.

[02-08-2006,16:00]
Anonymous
(in reply to: Globe and Mail Survey is BS)
Traditionally, university students and administrators don?t always agree, but on the second annual University Report Card, a survey of student satisfaction published in today?s Globe and Mail, there?s consensus that the survey?s methodology renders the findings unreliable.


"It?s frustrating," says Ashley Morton, president of the undergraduate Students? Administrative Council. "I?m not an expert but I have concerns about the methodology, especially when I see the same schools in the same city having widely different rankings in the category of opportunity for fun off campus. McGill is ranked #1 and Concordia is ranked #16 and they?re six blocks apart in Montreal. One of those numbers is incorrect so how many other numbers are unreliable?"


"Instead of a random sample, where all Canadian university students had an equal chance of being included in the survey, the Report Card uses a self-selected population, with the findings then applied to all students at the university," says David Farrar, vice-provost, students. "You can?t get a reliable picture of student satisfaction from this survey and that?s unfortunate because the more we know about our students, the more responsive we can be in our planning."

"The Report Card doesn?t reflect what students think," says Ranjini Ghosh, president of the Arts and Science Students Union (ASSU), which annually produces a student evaluation of arts and science courses and professors known as the Anti-Calendar. "In the Anti-Calendar survey, every arts and science student gets the chance to respond, while the Report Card was selective," she says. "Neither I nor the president of SAC heard anything about this survey until a month ago."

The Report Card survey sample consisted of some 26,000 students who logged on to studentawards.com, a website designed expressly for those seeking financial aid. The students were invited to answer more than 100 questions about student life on their respective campuses with the responses packaged into a ranking of those universities where more than 230 students responded. Only 38 of more than 93 universities and university colleges represented by the Association of Universities and Colleges of Canada were ranked in this year?s Report Card.

Farrar says that the student experience is a priority at U of T and is a major focus of the university?s emerging academic plan. The most rigorous look at student satisfaction in the university?s history is poised to begin this spring when U of T and seven other major Canadian research and teaching universities join some 600 U.S. universities in the National Survey of Student Engagement (NSSE). According to Farrar, the study is considered the authoritative benchmark on the quality of the student experience.

NSSE has been used in the U.S. for three years and has been refined to reflect current best-practice approaches to learning. Based on a random selection of first- and fourth-year students, the study asks them to rate their university on those approaches. It explores the breadth of student life including classroom experiences, relationships, course work, homework, student services and extra curricular activities. "It will help U of T identify where we?re succeeding and where there?s work to be done," says Farrar.

U of T already routinely conducts a number of student surveys and the findings reflect substantial satisfaction with U of T.

ASSU?s Anti-Calendar course evaluations look at more than 100,000 course evaluations. The ASSU student surveys consistently show that the average rating of U of T professors by the students who have taken their course is good/very good, with an average rating of 5.5 on a 7-point scale. When students were asked if they would have taken the course knowing what they know now, more than 70 per cent said yes."

In another survey, by the U.S.-based Higher Education Data Sharing Consortium, graduate students at U of T give very high marks to the quality of their academic program, and a solid majority is satisfied with overall student life. According to the survey, 92 per cent of U of T respondents rate their programs as good, very good or excellent, and 80 per cent would choose U of T again.

In the Office of Student Services, every department does an annual satisfaction survey and students rank all services very highly in terms of relevancy and responsiveness

"There?s a lot of choice at U of T, says SAC?s Morton, "in terms of courses, professors and clubs. We have more student clubs per student than most universities in North America. U of T?s size can be viewed as a problem to some, but that?s part of what makes U of T great. The university?s challenge is to make sure students know about all the options they have, but it?s also up to the students themselves to get out there and make connections. "Until I got involved," he adds, "I didn?t know all there was to do at U of T."

http://www.newsandevents.utoronto.ca/bin5/031015a.asp

[02-08-2006,16:01]
Anonymous



Canadian Universities Forum at Canada City Web Site | Start Home Based Business in Canada | Canadian and International FLP Business