Assignment #3 Program Evaluation

Planning a Program Evaluation for Bishop Murray High School

Manage the evaluation -Human Subjects Protection Timeline Responsibilities -Budget		1. Engage stakeholders
		2. Focus
		Describe program-logic model<!  Define purpose -Determine use/users -Determine key questions -Select indicators Determine design	3. Collect data
			Identify sources Select method(s) Pilot test Set schedule Determine sample	4. Analyze & interpret
				Process data Analyze Interpret data What did you learn? What are the limitations?	5. Use
					Share findings and lessons learned Use in decision making Determine next steps
Standards of evaluation: · Utility     · Feasibility     · Propriety     ·Accuracy

Stakeholders

Who should be involved? How might they be engaged?

The Stakeholders are the teachers, administration and students. The teachers and administration will be involved by completing a survey and being interviewed. Several will also be involved in providing input when analyzing the information collected. Several former and current students will also be interviewed. As well, there are many programs that involve using stakeholders from around the city, therefore, stakeholders also consist of the people who work the teachers and students of Bishop Murray. 

Focus the Evaluation

What are you going to evaluate?  Describe program (logic model). What is the purpose of the evaluation?

I am going to evaluate the graduation success rates of Bishop Murray’s First Nations Student Population. The goal is to determine if the programming offered is helping to improve the rates as is intended.

Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
Administration	To lead the staff to modify/adapt programs for the current needs.
Teachers	To help determine the supports that are needed for the students which could include further training
Superintendents	To provide the training and personnel needed To support possible changes to the existing building

What questions will the evaluation seek to answer?

Does the current programming work for First Nations Students? How is the programming implemented? Are activities delivered as intended? Are students being reached as intended? How can we improve graduation success rates? What other programming could be added or taken away?

 What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
What programs have been most successful for First Nations Students and why?	Questionnaires, surveys, interviews from teachers, former and current students, research of successful programs
What are the graduation rates of First Nations students?	Statistics, charts on file
What programs are least successful and why?	Questionnaires, surveys, interviews
How can we include the successful programs within Bishop Murray’s current building?	Results of least effective/most effective, re-evaluation of building, possible movement of school to a new facility

When is the evaluation needed?

The evaluation is needed by June of 2013 so that new programming can be put in place before the September 2013/2014 school year.

What evaluation design will you use?

This is a process and outcome based evaluation because I am seeking to determine if the program, as it stands, is being implemented as planned as well as if the program is indeed making a difference for First Nations Students. I plan to use Stufflebeam’s CIPP model because it is systematic and it looks at the program’s objectives as well as the overall impact on the students. By looking at the context of the program, it’s inputs, process and final product, I will be able to provide new suggestions that will delineate programs that are no longer working, and focus on the improvement of programs that are. Suggestions for new programming will be included.

The Tylerian Evaluation Approach will also be used as there is a definitive objective and goal. Actual documentation of the success rates will be used; both qualitative and quantitative measurements will be part of the data. Comparisons will then be made between the objective and the data.

         

Collect the information

What sources of information will you use?

Existing information:	Past statistics of graduation rates for First Nations Students.
People:	Long term staff members, students (former and current)
Pictorial records and observations:	Anecdotal notes, Observations of programs, Surveys, Interviews

What data collection method(s) will you use?

X	Survey	X	Document review
X	Interview		Testimonials
X	Observation	X	Expert panel
	Group techniques		Simulated problems or situations
	Case study		Journal, log, diary
	Tests		Unobtrusive measures
	Photos, videos		Other (list)

Instrumentation: What is needed to record the information?

I will use ‘Survey Monkey’ to create the survey, a digital recorder and transcriber for the interviews, a log to hold my anecdotal notes and observations, and recording sheets to track common themes of graduation rates and programs.

When will you collect data for each method you’ve chosen?

Method	Before program	During program	Immediately after	Later
Statistics  on First Nations Success Rates at Bishop Murray		X
Literature Review of Effective Programming for First Nations Students		X
Survey		X
Interview		X

Will a sample be used? 

No
Yes	X	If yes, describe the procedure you will use.
I will meet with one or more staff members to discuss the programming and use the information to help guide the questions for the surveys and interviews. Over several meetings, these questions will be modified with assistance from these teachers.

Pilot testing: when, where, how?

The survey and interview questions will be shared ahead of time so that teachers can provide their input. Editing and new questions will be made based on these conversations at an agreed  upon meeting place.

Analyze and Interpret

How will the data be analyzed?

The link to the survey will be sent to the staff via e-mail but the survey for the current students will be administered by me. I will also perform the interviews and digitally record them; a transcriber will be hired to compile the conversation. Next, I will chart common themes from the data provided. The statistics will also be evaluated and charted.

How will the information be interpreted—by whom?

Once the data has been compiled, I will meet with a small group of experienced teachers from Bishop Murray to evaluate the results. All input will be considered. Suggestions for improvement will be made.

What did you learn? What are the limitations?

From the evaluation, I will learn whether or not the current programming at Bishop Murray is making a difference in the graduation rates of First Nations students. I will also learn what is making a difference and what is not.

The limitations are that I, myself have worked within a similar structured program so there could be biases. The people who are involved in the evaluation all have an invested interest in the program, so looking to an outside evaluator to assist in the process could proved to be beneficial.

Use the Information

How will the evaluation be communicated and shared?

To whom	When/where/how to present
Superintendent	Written summary of results with charts, including recommendations – email -  June 2013
Administration	Written summary of results with charts, including recommendations – email June 2013
Staff Members	Informal Meeting with summary chart – June 2013

Next steps?

The next steps would include future planning for the 2013/2014 school year.

 Manage the evaluation                Standards

X	Human subject’s protection		Utility
	Management chart		Feasibility
X	Timeline		Propriety
X	Responsibilities		Accuracy
X	Budget

All participants will be given a pseudonym; their confidentiality will be protected.

The timeline for the evaluation is from January 2013 till it’s completion in June 2013.

Responsibilities for the evaluation fall on the evaluator, but the sharing of information will come from all stakeholders.