Features of Scan&Score
Scan&Score is a Web-based test scoring system that will:
- accept data files from external sources;
- provide methods for scoring, sorting and analyzing the input data;
- provide secure storage of results (2 years for active retrieval, 7 years archival storage);
- provide the ability for the user to retrieve the final analysis; and
- support the exam scanning and scoring services provided by the School of Medicine to the Medical School course directors.
Access and Designated Users
In order to gain access to Scan&Score, you must be a user in the system. Access is granted when you initially complete the scanning request form. If you later need to grant access rights to additional users (such as a designated teaching assistant), email your request to helpdesk@virginia.edu.
Scan&Score Instructions
Log into the Scan&Score application through NetBadge. You must ensure that popups are allowed in your Web browser.
Click the Select link next to the exam you'd like to work with.
From here, you can retrieve the raw test data, modify the answer key, score the exam, subset the exam by students or questions, and delete the exam. The exam owner may also give access to their exams to other users.
Modify the Answer Key
- Edit a question
- Allowing more than one answer
- Accept all answers as correct
- Change the weight of a question
- Exclude a question from the scoring
Subsets
A subset is a selection of questions or students from an exam. Subsets allow a professor or course director the ability to analyze smaller sets of data.
A subset by reference is such that when you modify the subset's answers you are also modifying the parent exam's answers.
A subset by copy is such that when you modify the subset's answers you are only modifying the subset's answers; the parent subset remains the same. A subset by copy is only available when subsetting by student.
Subset by questions can be used to see how the students did on questions relevant to a particular subject. For example, if questions 1–10 are testing students on material covered in class, and 11–21 are testing students on material from the text the professor could subset the exam into two sets and see if the scores vary between sets.
Subset by student can be used to break out different groups of students. If the class includes students who are taking the course as part of their major as well as students who are taking the course as an elective, the professor may want to compare the two groups to the grades overall.
By StudentChoose Reference or Copy.
Identify a group of student identifiers. The reports will only include data relevant to the identified group.
By QuestionIdentify a group of questions. The reports will only include data relevant to the identified group.
Available Reports
Test Scoring creates a number of reports for each test scanned. These reports can be viewed through the Scan&Score application.
- The Scores Report
A row per exam identifier showing the number of correct answers, value of correct answers and the percent correct for that student. Ordered by Respondent Id.
- The Score Frequency Report
Shows the distribution of the test scores for the class.
Sample Frequency Report for exam of 53 questions taken by 192 students. - The Score Ranks Report
Display the ranking of the student's test scores from highest to lowest. Total Correct is the correct number of questions answered. Total Points reflects the weight of the question.
Sample report of a 53 question test taken by 192 students. - The Item Analysis Report
An item analysis aids in determining which test questions were weak or represented unsatisfactory levels of understanding. The analysis includes for every question the question number, the correct choice, the number of omissions of that item, the numbers of persons marking each choice (shown on first row), the percentage of the class selecting each choice (second row) and a Pearson product moment correlation between each choice and total scores.
Results for the correct choice (marked by an asterisk) are most important. Good test questions have most or all of the following characteristics:- Between 30 and 85 percent of examinees should answer correctly. If the item is too hard or too easy it contributes relatively little toward ranking examinees according to their knowledge.
- Each of the answer choices should attract at least some of the examinees. If a wrong choice is so obvious that no one selects it, student testing time is saved by omitting it on future tests. There is no need to have the same number of choices for all items.
- The correlations between choices and total score should be positive (preferably .3 or higher) for the right choice and negative for the wrong choices. This outcome would indicate that better students tended to get the item right. If a positive correlation occurs for the wrong answer, it indicates that better students were misled into selecting the wrong choice. A poor item correlation can also alert the instructor to a possible mistake in filling out the key.
Sample Item Analysis Report shown above reflects an exam taken by 192 students consisting of 53 questions. - Error Listings (Wrongs) (currently not implemented)
Upon request, Test Scoring will provide an additional listing of each examinee's incorrectly answered items (including multiple marks). This report can eliminate the need to return Opscan sheets to examinees. The listing also indicates the number of omissions and the percentage of light marks.
Understanding the Test Output
Statistics, including number of students, number of questions means, standard deviations, high and low test scores and KR20 reliability estimate.
Median Number Right - the midpoint of the distribution where half the scores are above and half below.
Number of Students who took the test.
Reliability Estimate (KR-20) - an estimate of the correlation between the scores on the test and the scores which would be obtained if students were given a similar test on the same material. Satisfactory levels range from .5 for tests of less than 15 items to .8 and up for longer tests. The method for computing this estimate assumes that essentially a single area of knowledge is being tested. See Testing Memo 2 for more information on reliability scores.
Standard Deviation of Number Right - a measure of how spread out the scores are on either side of the mean number right.
Standard Error of Measurement - an estimate of how far a student's observed score may be from a true score, or the mean score that would be earned on many independent administrations of the test.
T-Score - is the number of standard deviations from the mean and may be used for "curving" grades.
For more information or for questions, contact the ITC Help Desk.