Test Score (ts) is an interactive program running on the RS/6000 computers that grades multiple-choice tests and produces a formatted report containing the scores, the answer distribution, and various statistics analyzing the scores and the test. It also has the option of producing a file of student IDs and test scores which can be used by another program, such as the UNIX gradebook package. (See U-021, ``An Electronic Gradebook Package.'')
Data that goes into ts is produced when UVa's standard answer sheets are optically scanned at Administrative Computing Services. These answer sheets require that there be no more than 240 questions on the test and that answers to multiple-choice questions be numbered 1 through 5. The scanned information is transferred via electronic mail or magnetic tape to an RS/6000 machine where ts resides.
Before running ts, take the answer sheets to Administrative Computing Services in Carruthers Hall. The answer key must be the first sheet read by the optical scanner, and it must have an empty or all zero Social Security Number (ID). The key must be followed by the students' answer sheets, each of which must have an entry in the ID field. Check each answer sheet to be sure an ID has been entered.
Data can be transferred from Administrative Computing to your account on an RS/6000 using either electronic mail or magnetic tape. We recommend that you use electronic mail. Not only will this save you time carrying tapes back and forth between computing centers, but using tapes requires a fairly good understanding of transferring files from tape to disk and from one machine to another. If you use tapes, you must also have an account on watson, the IBM 3090 mainframe, in addition to your RS/6000 account. To get an account on either machine, request a form from the greeter at the Help Desk in Room 235 in Wilson Hall.
To transfer data to your RS/6000 account using electronic mail, follow these steps:
1. Request that Administrative Computing electronically mail you the scanned results, which contain the students' IDs and answers. The form that you fill out when you take the answer sheets to be scanned will have a place for an electronic mail address. It is best to have the scanned data mailed directly to the RS/6000 machine on which you will run the test scoring program.
2. If you normally use mail on a machine other than the RS/6000s, you may receive files from Carruthers Hall on your primary mail account instead of your RS/6000 account. If this happens, forward the message containing the data to your full address on the RS/6000. For instance, if you received the data on your PC Mail account and your RS/6000 account (xyz8r) is on avery, forward the message containing the data to xyz8r@avery.med.virginia.edu. Then use the mailer on the RS/6000 machine to read the data into a file.
3. Save the results from the mail program into a file. Use an editor to remove the mail headers from the beginning of the file and any extra information from the end of the file. There should be four lines of data per student.
4. Type the following command to prepare the data for ts:
where mail_file is the file containing the data that was mailed to you, and answers_file is in the format required by ts.
Begin the test scoring program by typing
The program will begin by asking you for the following things:
Enter 0 for red (old) or 1 for blue (new):
How many multiple choice answers are there?
Enter 4 or 5:
Enter the number of questions on the test (1-240):
Enter the name of file containing the scanned data
(student IDs and answers):
Formatted output from this program will be saved into a file which you can send to a printer at the conclusion of this program.
What NEW name do you want to give this file?
If you are using a gradebook program to manage your grades, ts gives you the option of creating a file of IDs and test scores that other programs can read. For instance, the gradebook program which runs on the RS/6000s can be used to read this file of student scores and add them to an electronic gradebook. If you want to create this file, enter a file name that is not longer than twenty characters. The file name should not already exist in your current directory. The format of this file is shown in the Appendix at the end of this document. The program will continue:
Output from this program can be saved to a file containing student IDs and scores to be used by other programs.
Do you want to create this file? (y or n)
Give this file a name (up to 20 characters). This name should not already exist in this directory.
The test scoring program will then ask you for these things:
Enter the total number of points for this test.
Adjusting grades gives students a penalty for guessing. If grades are not adjusted, the point value of a question is subtracted for a wrong answer. If grades are adjusted, the penalty for a wrong answer is calculated as follows:
where adjustment = 5 (the number of choices) divided by (5 - number of correct answers)
If full credit is given for any correct answer regardless of how many correct answers there are, then the adjustment = 5/4 . The ts program will continue:
Do you want to adjust the grade for wrong answers? (Answer y or n)
Adjusting grades for wrong answers makes it possible for students to have negative grades. If you indicate that you want to adjust grades, you will be asked:
Do you want to set negative grades to 0? (Answer y or n)
Example
On a 20-question test worth 100 points, a student missed only one of the 5-point questions. The question had five choices, but only one of the choices was the correct answer. Adjusting the grade for wrong answers gives:
The grade would have been 95 had grades not been adjusted. More examples illustrating grade calculations can be found at the end of this document.
If a question has more than one correct answer, you can tell the program how you would like the question to be graded. You can either give full credit for any correct answer, or calculate the credit using the point value of the question divided by the number of correct answers. The program will continue:
Do any questions have more than one correct answer?(Answer y or n)
Do you want full credit for any correct answer?(Answer y or n)
If you answer n, the amount of credit will be divided by the number of correct answers. If you answer y, full credit will be given for any correct answer.
Weight factors, sometimes called scale factors, allow you to assign different point values to each question. If you do want different weights for questions, decide how much each question is worth compared to the others. Then group the questions so that consecutive questions with the same weight are together. The number of groups will be the number of weight factors you will need to enter at the prompt:
Do you want weight factors? (Answer y or n)
Enter the number of weight factors
(no more than the number of questions).
Now tell the program the weight factors and questions to which they apply. You will first be asked the number of the last question to which the weight factor applies, then asked for the weight factor itself. This is best shown in an example.
Example
You give a 20-question test, for which there are a total of 100 points. The first 10 questions count twice as much as the second 10 questions. Since there are two sets of consecutive questions with the same weight, you would want two weight factors. All questions have only one correct answer, and no adjustment is made for wrong answers. The interaction should be as follows:
What is the number of the last question to which the next weight factor applies? 10
What is the weight factor for questions 1 to 10? 2
What is the weight factor for questions 11 to 20? 1
These equations show how to calculate the point value of each question:
If a student missed only question 20, his grade would be calculated as follows:
Special conditions allow you to change answers on the key, delete questions from grading, and change weight factors. The program will continue:
You must now specify which special conditions you want.
You will be able to change an answer on the key, specify that more than 1 answer is correct, delete a question from grading, or change a weight factor. Give the number of the question for which a change is to be made.
Enter 0 for no changes or the question number:
If you do not need to make any changes to the answer key, enter a zero (0) and the program will continue. If you enter a question number instead of a zero, this menu will appear:
You can now:
1 = change answer on key
2 = two answers correct
3 = three answers correct
4 = four answers correct
5 = all answers correct
6 = delete question from grading
7 = change scale factor
Which change do you want to make?
If the answer to the question number entered above was wrong on the key, use option 1 to change the answer on the key. Use options 2, 3, 4, or 5 if more than one answer was correct for the question. Options 1 through 4 will prompt you for the correct answers to the question:
What is the letter of the Nth correct answer? where Nth is 1st, 2nd, 3rd, or 4th.
Option 6 allows you to delete a question from grading. The question whose number you entered will be completely ignored when calculating students' grades on the test.
Option 7 allows you to change the scale factor (weight) for a question. If you choose this option, the program will ask you:
What is the new weight factor for question N?
After you have entered everything the program needs for the option you selected, you will see the question:
Are there any more questions with special conditions?
(Answer y or n)
Answer y here if you want to make any more changes, or n if you do not.
Example
On your answer key you had marked A as the correct answer to question 9, but you now realize both A and B are correct answers. To correct this situation, proceed as follows:
Give the number of the question for which a change is to be made. Enter 0 for no changes or the question number. 9
You can now:
1 = change answer on key
2 = two answers correct
3 = three answers correct
4 = four answers correct
5 = all answers correct
6 = delete question from grading
7 = change weight factor
Which change do you want to make? 2
What is the letter of the 1st correct answer? A
What is the letter of the 2nd correct answer? B
Are there any more questions with special conditions?
(Answer y or n) n
Once all questions have been answered, ts will do the actual grading. When it is finished, you will see a message stating that the formatted file and the optional scores file have been saved in your current directory with the names that you entered.
To print the formatted report on an Apple LaserWriter PostScript printer on UVa's network, type the command:
where printer_name is the name of the printer, and formatted_file is the name you gave the formatted report.
To print the optional file that contains only IDs and scores on an Apple LaserWriter PostScript printer, type the command:
where printer_name is the name of the printer, and results_file is the name you gave the file of IDs and test scores.
The formatted report includes these features:
The labels on the histogram represent a range of numbers, so they may not represent an actual score.
If everyone got the question right, the index=100%; if everyone got it wrong, the index=0%. The index is computed using the 27% best scores and the 27% worst scores as follows:
where TOP is the distribution of scores for the best 27% of student scores; BOT is the distribution of scores for the worst 27% of student scores; NTOP is the number of students in the top 27%; NBOT is the number of students in the bottom 27%.
This index is computed by comparing the answer given by the top and bottom 27% of the students for each question. The index is larger when the top students get the question right and the bottom students get it wrong. If the index is more than about .25, then the question was harder and discriminated between good and poor students in terms of their performance on this test. The index is calculated as follows:
where TOP, BOT, NTOP, and NBOT are defined above.
A 20-question test is given, for which there are a total of 100 points. Questions 1-10 have a weight factor of 2.0, and questions 11-20 each have a weight factor of 1.0.
1. Assuming there is only one answer per question, if a student misses only question 20 and adjustment is made for wrong answers, the calculations are as follows:
total weight factors = (2.0 * 10) + (1.0 * 10) = 30
point value of question 20 = 1 * 100/30 = 3.33
penalty = 3.33 * 5/4 = 4.16
score = 100 - 4.16 = 95.84
2. If question 1 of this test has two correct answers, A and B; full credit is not to be given for any correct answer; and adjustment is made for wrong answers; then the calculations are as follows:
total weight factors = (1.0 * 1) + (2.0 * 9) + (1.0 * 10) = 29
Thus, if a student missed only question 20:
penalty = 3.45 * 5/4 = 4.31
score = 100 - 4.31 = 95.69
However, if a student missed only question 1, his score would be lower, because there is a 2-in-5 chance of getting the correct answer rather than a 1-in-5 chance:
penalty = 3.45 * 5/3 = 5.75
score = 100 - 5.75 = 94.25
The format of UVa's standard answer sheet was changed in late 1991 so that it can be used for several different types of questionnaires. The top part of the new form has room for several more characters than the old, thus extending the length of each line put out by the scanner from 284 characters to 288 characters. In order to distinguish between the two forms, the color was changed from red (on the old form) to blue (on the new form). The ts program needs to know the color of the form, so that it can correctly determine the placement of each answer.
In order to e-mail a file from Administrative Computing in Carruthers Hall, the length of each line must be no longer than 72 characters. Therefore, each record from the scanner is split into four lines for mailing. The format of the files from both forms is shown below:
____________________________________________________________________________ | New (Blue) Form || Old (Red) Form | |____|________|_______________________||____|_______|_______________________| |Line|Columns | Contents ||ine |Columns| Contents | |____|________|_______________________||____|_______|_______________________| | 1 | 39-47 |Social Security Number ||1 | 35-43 |Social Security Number | |____|________|_______________________||____|_______|_______________________| | 1 | 49-72 |Questions 1-24 ||1 | 45-72 |Questions 1-28 | |____|________|_______________________||____|_______|_______________________| | 2 | 1-72 |Questions 25-96 ||2 | 1-72 |Questions 29-100 | |____|________|_______________________||____|_______|_______________________| | 3 | 1-72 |Questions 97-168 ||3 | 1-72 |Questions 101-172 | |____|________|_______________________||____|_______|_______________________| | 4 | 1-72 |Questions 169-240 ||4 | 1-68 |Questions 173-240 | |____|________|_______________________||____|_______|_______________________|
The command gconcat concatentates these four lines back into one record which is the format required by ts:
____________________________________________________________________________ | New (Blue) Form || Old (Red) Form | |__________|__________________________||__________|_________________________| | Columns | Contents ||Columns | Contents | |__________|__________________________||__________|_________________________| | 39-47 | Social Security Number || 35-43 | Social Security Number | |__________|__________________________||__________|_________________________| | 49-288 | Questions 1-240 ||45-284 | Questions 1-240 | |__________|__________________________||__________|_________________________|