StudentsReview's OFFICIAL Rankings are the first rankings to be COMPLETELY generated from student opinion, and to publicly publish their ranking methodology & analysis. There are no preconceptions, university administrators, group consensus, or personal expectations governing our rankings. We perform an analysis that is transparent and understandable, and hopefully informative, governed only by tabulated student opinion. 
Sophistication of analysis is reflected by full and transparent description of internal biases, and its “apples to apples” analysis.

This document will explain how the data was analyzed, how the StudentsReview ranking system operates, and how “apples to apples” comparison is achieved.  If you have a quick question, head on over to the 2005 NYD FAQ (Frequently Asked Questions) page.

The StudentsReview Ranking system consists primarily of filtering machinery and a public formula, which is described below.  There is a minor component to the math (and we assure you, it is only math connected with analysis of student surveys) that we are keeping private to prevent exploitation. If public, it could be exploited by third parties who want control the rankings, and it would be impossible to model the data, preventing any meaningful analysis in the future. 

Table of Contents
Purpose
Data Purity
Filtering
Biases
Sampling Biases
Apples to Apples
Private Component
Limitations
Contentions
Conclusions
FAQ

For this particular set of rankings, and in general, StudentsReview takes the approach that the quality of education will surface and provide both confidence and opportunities in the workplace once the tangible skills are revealed. While REPUTATION is fairly well known and understood, the quality of education at any particular institution is an unknown that is insightful to reveal.

Data Purity
First and foremost, StudentsReview considers itself a data analysis company, and so to the best of our ability, StudentsReview's data is clean of unknown/3rd party biases. We do not modify or remove surveys to “craft” a particular image. If data is not managed with integrity, then any conclusions drawn from it are meaningless and misleading.

Several institutions have led surveys from designated “feeder” students — to bias our data in favor of their institution.  Apart from the data we've been able to detect ourselves, the students have surprised us to reveal that they are “self reporting” — that is, that they have acted as ethics police, reporting directly to us those incidents of bad-practice. Resultingly, we dropped all of the affected surveys and performed manual verification.

A number of institutions have sought to leverage threats of legal action, or a business relationship to influence us into removing or altering data. In every contention, except those where the data was found to be genuinely invalid, the demands and solicitations were denied.

Filtering
Data is statistically filtered prior to ranking to remove invalid, duplicate, or statistically blank surveys.  Invalidity is determined by a statistical algorithm which looks for surveys that are not self-consistent, or reflect a systematic corrupting effect on the data, either accidentally or with intent.

5,000 valid surveys were analyzed statistically, and a gaussian matrix was created to model the survey patterns within and between surveys.  We can now identify those surveys that: vary too little, vary too much, have fields that do not covary properly, or are inconsistent.  (i.e.  rating the university as an A for Friendliness, but then rating Social life an F).  The filter was then trained on a marked set of valid and invalid data to set its thresholds for how much inconsistency, variance, and survey-survey interaction it can tolerate.
A rule-base system was also created to identify duplicates and model trends of surveys from the same machine. This allows us to be able to identify if a person is falsifying many surveys.  FFT analysis is employed to determine the “data content” of each survey as well, providing more information for modeling. 

The combination of the trained statistical filter, correlation matrix, and survey model runs autonomously, applying its decisions uniformly to all surveys from all schools.  For the OFFICIAL rankings, the thresholds for inconsistency were tightened a bit to prevent spurious or anomalous data from affecting the rankings too heavily.

Out of an early sampling of 7,500 undergraduate student surveys, 483 surveys were rendered invalid. Inspection of the invalid surveys revealed a failure rate of 5% and 7% for good and missed surveys respectively.  (24 of the 483 surveys were actually “good”), then 32 invalid surveys were missed, leading to an overall incorrectness failure of about 1%.

Biases
There are ALWAYS biases in ANY data gathered by ANYONE.  It does not matter who is doing it, when, what methods are used — there are ALWAYS biases in the data. Anyone who claims their data is unbiased is outright lying.  The most important thing is to understand what those biases are.  Because StudentsReview's surveys are gathered only though our website, with no financial or contest “reward” (we feel it cheapens students' opinions), students who take our survey are self-selected, and like-mindedly referred.  This leads to two predominant results.  The first is a natural bias in our data that is often disproportionately criticized for being overly negative or bitter.  Many do not realize that 61% (8875/14538), the vast majority of our surveyors have said that they would choose to return to their institution given the chance, so the self-selected bias is not as negative as critics have expected. 

If anything, the self-selected bias is actually a polarizing one, in that those students who feel positively tend to “reply” in the comments to those who feel negatively, creating what appears to readers to be a strongly partisan opinion of the school on our website.  But even that belief is incomplete, because many of the students who do not write a comment after taking the survey are those without strong opinions.  So the survey data itself (not the comments) tends to have a reasonable mix of positive, negative, and middle of the road opinion.

The second result is a positive one.  The same self-selected students are also the ones with time to sit down and expound in-depth about their experiences — at far greater detail than possible with any on-campus surveying.  As such, much more meaningful causal insight is gained about they physical factors leading to any satisfaction or dissatisfaction than any short quotes could provide.

Finally, it has been brought to our attention that some critics believe surveyors are fulfilling some “ulterior” motive by taking our survey (i.e.  deferring applicants to reduce competition, etc.  ).  Except in the cases of university admissions bad practice (described above), there is little to no reward or consequence that student can achieve by taking our survey.  That is, deferring prospective applicants through our site as a personal tactic would not achieve visible results for several years — most students will have graduated by then — it is long after most peoples' reward horizons.  It is impossible for them to achieve any ulterior motive beside informing prospective students.

Sampling Biases/Representation 

Figure 1.  - The disparity between the survey reponses (biases) at two very different institutions: Pensacola Christian College (PCC) and the Massachusetts Institute of Technology (MIT)

The most common concern raised about our rankings is one of representative sampling — that the sampling of students in the surveys is not representative of the true student opinion.  For instance, consider if more “positive” students or more “negative” minded students than the school's average take the survey at any particular school (Figure 1.).  Those surveys could bias and lead to a school rank that is artificially high or low. 

A common method to overcome this problem is to model the shape of biases across ALL surveys and ALL schools (Figure 1 - black line).  That shape is used to neutralize the response biases in any particular school, so that all schools have the same modelled sample. Unfortunately, to some degree, neutralization is a bad thing — consider a school where most of the students actually ARE dissatisfied, or the educational quality really IS NOT that stellar. Neutralization would disproportionately suppress the valid (dissatisfied) opinions, and overly amplify the one or two satisfied opinions — leading to an artificially high score.

The other method, and the preferred one, is to simply acquire more surveys.  As the numbers grow, the sampling converges on a representative sampling.

Just to reiterate, 61% of our surveys are positive, so we do not maintain an overly negative bias.

Free Variables, Normalization, and Synthesis
( “Apples to Apples” )

Regardless of what anyone may say, schools are inherently non-comparable. They have different student bodies, different majors, different educational offerings, different locations, weather, and ultimately different people doing the rating. Ranking the schools in any fashion is the equivalent of me giving you an orange, and Joe over there an apple, and asking each of you, “how sweet is that fruit?”.  You might say, “This orange is super-tangy and sweet”, and Joe might say, “My apple is really tart and sweet!”.  Now which fruit is sweeter?  Who knows?  Two different people said something about two very different fruits — we don't know if Joe is more or less sensitive to sugar than you are, if the apple actually has more or less sugar content, or if the multitude of other flavors teasing your taste buds interfere in some way. The point is, you are two different people looking at two completely different fruits.

We overcome this problem in three steps: Binding Free Variables, Synthesis, and Normalization. But before diving in, it is useful to provide an overview of what is occurring.  Essentially what we do is break apart what we know about you and Joe into as many factors as possible — are you male, how much do you like fruits, etc.  Then we look at what we can learn from a large number of people like you and Joe.  How similar are your sensitivities to sugar, how do the similar people to Joe rate oranges, and how do the similar people to you rate apples?  How do you covary? Using what we know about how you both are related, we synthesize a kind of “ghost” next to each of you to act as a “stand in” for the other person.  True, it is not the same as if Joe had tasted an orange himself, or if you had tasted an apple, but it provides a suggestion of how you “might have” rated it, if you did. 
Finally, we normalize, so that the same number of Joes and the s