Blog 4

RETURN ON INVESTMENT


"Even if you are on the right track, you'll get run over if you just sit there."
--Will Rogers--


Training For Dummies (ISBN 0-7645-5985-0) - suggested reading


Image can be quantified


Example: the Tylenol tampering incident. Proper intervention by full recall of products saved the company's reputation and enhanced trust in the product. The sealed caps and full accountability stance led the industry, justifying the investment.



Business impact can be measured


Projection, forecast, prediction: estimated benefit of intervention


Loaded salary: base salary plus benefits


A dissatisfied customer will tell 28 people about their bad experience


Customer mortality costs affect the bottom line


Customer attraction costs improve the bottom line


Intangibles can be quantified

Courtesy of Dr. Ackerman

Benchmarking - using another company's product quality as a goal

Likability factor - hard to quantify


Example: Nordstrom's customer service policy - 'the customer is always right'
The store allowed the return of a car tire that was not purchased at the store in exchange for store credit. They then tracked the store purchasing of the individual and family. The amounts purchased far exceeded the amount of the exchange plus a repeat customer was gained, justifying the investment.

Practice Exercise - ROI

Columbus Sulpher Company considered implementing a two-day workshop to decrease data entry errors of Form #1492. By reducing rework of data entry, Dr. Cristopher estimated an annual savings of $40,000.

An outside vendor offered to sell Columbus Sulphur Co. the two-day workshop materials and usage rights for $10,000, for one year only. Costs for delivery, loaded salaries of attndees, and other miscellaneous expenses would be $6000. The total costs of the one-year program investments would be $16,000.

Before making a decision on whether they should purchase the workshop for one year, upper management requested cost-benefit projections.
_______________________________________________________
Calculate the following:

ROI Annual Savings $40,000
Average Investment 16,000 %


Cost/Benefit Total Benefits $40,000
Ratio Progam Cost 16,000 Ratio

Payback Total Investment $16,000
Period Annual Savings 40,000 Yr Months
_______________________________________________________

Companies will look at the bottom line - return on investment - only, unless they are in a position where there is a problem and the other evaluation level can be more helpful

Some companies are getting away from using evaluation level one

Experimental Bias - the favoring of certain outcomes over others

Mandated Training - when training is mandated to the learner, instructors should be sensitive to this and take measures to prepare the student for instruction by identifying and accomodating student needs for specific training, hours, break periods, etc.

Measure and quantify proposed changes to the organization

90 Second Evaluation
This course was very interesting and also very applicable to my job. I like that the course was about application of the subjects learned. The lessons I found to be the most useful to my learning of assessment were:

1. The ADDIE model. Though I had a brief intro to this in the first course, it was good to go back and do a thouough review and expansion on this.

2. The levels of assessment. This was an entirely new concept that I felt filled in some of the blanks and added to the ADDIE model.

3. The levels of evaluation. I also had a brief intro to this, but this course expanded my knowledge of this topic and explained in detail how it is applied. I appreciate the application focus and extra time spent on this.

4. New concepts. The Pygmalion effect, the Hawthorne effect and the Primacy Recency effect just to name a few were outstanding to learn. Real college stuff.

5. Building rubrics. This is something that I feel is outstanding because it standardizes evaluation at a deeper level than just a checklist.

"Chaotic action is preferable to organized inaction."
--Will Rogers

Blog 3

QUIZ-MAKER PARTNER PROJECT

The project required building a rubric which proved to be very interesting and challenging. It required a lot of detail and precision to make sure each category aligned with each other across the scoring columns. It forces you to evaluate your process and determine what level of performance is excellent and what level is unacceptable. It's value is that it provides a clear and detailed means of assessing something. It could be used to assess almost anything.




To rate your quiz-maker using the rubric you had to have a thorough understanding of the features of the quiz-maker.


Making the test was more detailed than I thought. I changed my direction on what type of test I wanted to make a couple of times based on the information I was making the test around and the types of questions I was able to form from it. I was happy with the final product. From making the test, I began to see how the test matches up with the experimental design concepts. That dictated the level of difficulty I tried to place in the questions and who I sent the test to in an attempt to influence the results.

I sent my test out to quite a few people to insure that I received enough responses to the test. I sent them to instructors at work, to SIU students, and to family members. I wanted a varied grouping to see which groups had the best scores = background with the material (or possibly good guessers).

I received positive feedback for the report generator of the testmaker. The certificate was a nice touch. The AF members that took the test could appreciate the automatic scoring feature. Many would have had to retake the test after seeing that they did not achieve the minimum score like our annual training CBTs.

My project partner will each create a summary of the project and will swap with our sister project partner to provide/receive the classmate peer review.

Blog 2

WED 463 - Evaluation Levels

40-60% of the ISD process is spent in design

Kaizen - continuous improvement, a ground-up improvement system

On-line instruction - for every one hour of delivery, 100 to 300 hours of development is required

Formal evaluation very critical to on-line instruction

Compound objective - more than one performance action required for one obj
(use only one performance per objective)

Mager Three-Part Objective
1. Performance that states what the learner is expected to do

2. Conditions describe the circumstances under which the performance will occur

3. Criterion states the level of quality and/or competence that must be reached or surpassed

Important: measurement of performance and role of objectives are continuous throughout the model

F ormative
E valuation
E ach
D esign
S tage

Survey mortality rate - the level at which a survey is not completed and the factors that affect non-accomplishment of parts of the survey

Primacy/recency effect - the first and the last items read are remembered, in a paragraph, the first word and the last sentence is recalled, break up long classes into segments with more beginning and endings to take advantage of this effect

Miller's magic number - Working memory is limited to only 5 to 9 numbers (7+or- 2) that can be calculated at a time (7 for digits, 6 for letters, 5 for words)

Chunking - elements of data in chunks to enhance short term memory

Rubric - a rating scale, evaluation by classification of numbers

Experimental Design
Measures impact of training

Safety, legal or public service must receive the required training

On-the-job can increase learning to pass tests

Random sampling - personnel for control groups must have equal chance for selection

Random number tables - avoid experimental bias

Sometimes learners learn from the pre-test and are desensitized

Internal Validity - whether observed changes can be attributed to your program or intervention and not other possible sources

Design of experiments - design of information-gathering exercises
where a variation is present

Survey Types
Questionaire - research instrument consisting of a series of questions and other prompts for the purpose of gathering information

Interviews - more personal means of gathering information, can probe or ask follow up questions

Blog 1

WED 463 - Assessment of Learning Performance

ADDIE - Instructional systems design model
Anaylyze, Design, Develop, Inplement, Evaluate

Formative evaluation - to collect data througout instruction to see if instruction is doing what it is designed to do, is student performing as intended

Donald Kirkpatrick - 4 levels of evaluation
1. Reaction - Non performance, feelings, attitude measurement, opininon, not considered best
data
2. Learning - Pre-post tests, assessment, perform in class (controlled environment)
3. Transfer - Performance in field, application, self appraisal
4. Business Impact - 40% of US worker time is rework - errors are caught 'down the line',
in the Japanese TPS employees can stop the line when an error is detected

Jack Phillips - 5th level of evaluation
Return on Investment - every complaint can be converted to a dollar sign, measures time and money, 5% of US companies perform an ROI assessment, measure for every dollar spent how much is returned

Train personnel to proficiency performance as soon as possible to increase productivity

Performance Gap Analysis
What is - current state Need - discrepancies What should be - desired state

Determine intervention - training or non-training, informaton, or motivation

Most effective way to fill the gap - cost, time, etc.

If the workers can perform under duress, skills are present - motivation is lacking

When pressure does not work, training may be the problem

Use a run chart to calculate data over a long period to identify anomalies

Test Making
Be unbiased when collecting data, be careful about variables that effect people's behavior

Hawthorne Effect: Aptitude does not predict performance

Pygmalion Effect: Once an expectation is set, we act in ways consistant with the expectation

Rapid Prototyping: Solid free-form fabrication of 3 dimensional objects w/CAD

Evaluation Myths
What is not measured, does not get done

Formative evaluation - measuring progress throughout the training process

Training - Instruction to improve skills for job performance (must include practice, feedback, and application) (Need to know information)

Education - General, academic (Nice to know information)

Presentation - Stating information (Spray and pray)

Organization infrastructure can also affect motivation to perform - policies, equipment, time lines, etc

Gap analysis - Collect data to fill the gap (observation, asking, surveys, interviews to get information (an intervention must be targeted at the right fix or problem to be effective)

Four Types of Assessments
1. Pre-test - Beginning of course measurement
2. Pre-requisites - Required before block of instruction
3. Post Test - After course measurement
4. Embedded - Measuring practice during the course, feedback, quizzes

Halo Effect - Positive or negative assumptions based on past performance