Blog 3

QUIZ-MAKER PARTNER PROJECT

The project required building a rubric which proved to be very interesting and challenging. It required a lot of detail and precision to make sure each category aligned with each other across the scoring columns. It forces you to evaluate your process and determine what level of performance is excellent and what level is unacceptable. It's value is that it provides a clear and detailed means of assessing something. It could be used to assess almost anything.




To rate your quiz-maker using the rubric you had to have a thorough understanding of the features of the quiz-maker.


Making the test was more detailed than I thought. I changed my direction on what type of test I wanted to make a couple of times based on the information I was making the test around and the types of questions I was able to form from it. I was happy with the final product. From making the test, I began to see how the test matches up with the experimental design concepts. That dictated the level of difficulty I tried to place in the questions and who I sent the test to in an attempt to influence the results.

I sent my test out to quite a few people to insure that I received enough responses to the test. I sent them to instructors at work, to SIU students, and to family members. I wanted a varied grouping to see which groups had the best scores = background with the material (or possibly good guessers).

I received positive feedback for the report generator of the testmaker. The certificate was a nice touch. The AF members that took the test could appreciate the automatic scoring feature. Many would have had to retake the test after seeing that they did not achieve the minimum score like our annual training CBTs.

My project partner will each create a summary of the project and will swap with our sister project partner to provide/receive the classmate peer review.

Blog 2

WED 463 - Evaluation Levels

40-60% of the ISD process is spent in design

Kaizen - continuous improvement, a ground-up improvement system

On-line instruction - for every one hour of delivery, 100 to 300 hours of development is required

Formal evaluation very critical to on-line instruction

Compound objective - more than one performance action required for one obj
(use only one performance per objective)

Mager Three-Part Objective
1. Performance that states what the learner is expected to do

2. Conditions describe the circumstances under which the performance will occur

3. Criterion states the level of quality and/or competence that must be reached or surpassed

Important: measurement of performance and role of objectives are continuous throughout the model

F ormative
E valuation
E ach
D esign
S tage

Survey mortality rate - the level at which a survey is not completed and the factors that affect non-accomplishment of parts of the survey

Primacy/recency effect - the first and the last items read are remembered, in a paragraph, the first word and the last sentence is recalled, break up long classes into segments with more beginning and endings to take advantage of this effect

Miller's magic number - Working memory is limited to only 5 to 9 numbers (7+or- 2) that can be calculated at a time (7 for digits, 6 for letters, 5 for words)

Chunking - elements of data in chunks to enhance short term memory

Rubric - a rating scale, evaluation by classification of numbers

Experimental Design
Measures impact of training

Safety, legal or public service must receive the required training

On-the-job can increase learning to pass tests

Random sampling - personnel for control groups must have equal chance for selection

Random number tables - avoid experimental bias

Sometimes learners learn from the pre-test and are desensitized

Internal Validity - whether observed changes can be attributed to your program or intervention and not other possible sources

Design of experiments - design of information-gathering exercises
where a variation is present

Survey Types
Questionaire - research instrument consisting of a series of questions and other prompts for the purpose of gathering information

Interviews - more personal means of gathering information, can probe or ask follow up questions

Blog 1

WED 463 - Assessment of Learning Performance

ADDIE - Instructional systems design model
Anaylyze, Design, Develop, Inplement, Evaluate

Formative evaluation - to collect data througout instruction to see if instruction is doing what it is designed to do, is student performing as intended

Donald Kirkpatrick - 4 levels of evaluation
1. Reaction - Non performance, feelings, attitude measurement, opininon, not considered best
data
2. Learning - Pre-post tests, assessment, perform in class (controlled environment)
3. Transfer - Performance in field, application, self appraisal
4. Business Impact - 40% of US worker time is rework - errors are caught 'down the line',
in the Japanese TPS employees can stop the line when an error is detected

Jack Phillips - 5th level of evaluation
Return on Investment - every complaint can be converted to a dollar sign, measures time and money, 5% of US companies perform an ROI assessment, measure for every dollar spent how much is returned

Train personnel to proficiency performance as soon as possible to increase productivity

Performance Gap Analysis
What is - current state Need - discrepancies What should be - desired state

Determine intervention - training or non-training, informaton, or motivation

Most effective way to fill the gap - cost, time, etc.

If the workers can perform under duress, skills are present - motivation is lacking

When pressure does not work, training may be the problem

Use a run chart to calculate data over a long period to identify anomalies

Test Making
Be unbiased when collecting data, be careful about variables that effect people's behavior

Hawthorne Effect: Aptitude does not predict performance

Pygmalion Effect: Once an expectation is set, we act in ways consistant with the expectation

Rapid Prototyping: Solid free-form fabrication of 3 dimensional objects w/CAD

Evaluation Myths
What is not measured, does not get done

Formative evaluation - measuring progress throughout the training process

Training - Instruction to improve skills for job performance (must include practice, feedback, and application) (Need to know information)

Education - General, academic (Nice to know information)

Presentation - Stating information (Spray and pray)

Organization infrastructure can also affect motivation to perform - policies, equipment, time lines, etc

Gap analysis - Collect data to fill the gap (observation, asking, surveys, interviews to get information (an intervention must be targeted at the right fix or problem to be effective)

Four Types of Assessments
1. Pre-test - Beginning of course measurement
2. Pre-requisites - Required before block of instruction
3. Post Test - After course measurement
4. Embedded - Measuring practice during the course, feedback, quizzes

Halo Effect - Positive or negative assumptions based on past performance