Skip to main content

Observation should inform program evaluation

By Samantha Grant

Have you ever watched a youth program where everything seemed to be working? As a youth worker, your gut reaction can be a good gauge of when things are "clicking" inside youth programs and when things need improvement. Sometimes with the current pressure to show the outcome and impact of our programs, we lose sight of the skills we develop through experience in youth work - our ability to observe and assess.

Observational methods in evaluation or research are gaining popularity in school and youth settings. In Minnesota 4-H, we have been investing in the Youth Program Quality Assessment. This standardized observational tool allows youth workers to assess safe environments, supportive environments, interaction, and engagement. There are many other tools for assessing youth program quality. Check out The Forum for Youth Investment for a review of tools.

An article in the spring 2011 Afterschool Matters publication takes a look at the Self Assessment of High-Quality Academic Enrichment Practices. Holstead and King detail the growing emphasis of self-assessments inside 21st Century Community Learning Centers. Their article gives a glimpse into aligning self-assessment with standards of program practice and highlights the pros and cons of self-assessment. They note the power of self-assessment for providing information that can build "programs that provide the best possible services to participants."

Pros of self-assessment include: it encourages staff to be reflective, it promotes continual reflection, and it can generate important feedback that staff can use. One of the biggest cons of self-assessment is the risk that in tailoring tools to fit your program, you can lose the reliability and validity of the instrument.

I am a huge proponent of assessing the quality of our learning environments and I strongly believe in observation. Sometimes that means using a standardized tool, like the YPQA, but sometimes it means creating a tool that hones in on what is important in your organization. It can also mean just stopping to watch what is happening inside your program.

So what can observation add to your program?
  • Observation prompts program staff to slow down and be reflective
  • Observation takes you to the heart of youth programs - the point of service - where adults and youth come together
  • Making observation part of your practice helps to build skills in youth workers and encourages a climate of dialogue and improvement

Are you a proponent or practitioner of observation as part of program self-assessment? Why or why not?

-- Samantha Grant, evaluation director

You are welcome to comment on this blog post. We encourage civil discourse, including spirited disagreement. We will delete comments that contain profanity, pornography or hate speech--any remarks that attack or demean people because of their sex, race, ethnic group, etc.--as well as spam.

Print Friendly and PDF

Comments

  1. Sam - I am both a proponent and a practitioner of observation as part of program self-assessment. While it is great to have the resources to have an external observer, much can be gained by taking a birds-eye view as an insider to a program. In my experience, it helps to know the purpose for the observation and a clear idea about how the observations will inform your work ahead of time. You uncover the two broad functions that can be served by observation - the more structured task of looking FOR (did this important behavior occur or not occur while I observed and with whom?)and the function of looking AT (what did I notice that seems important to what is working or not working in this program?). Observation can be a challenge for those of us who are teachers at heart. As one teacher-turned-observer stated, "It is hard to stand by and watch when I see young people who I could help learn!" Sam and others, what skills do we need to hone in order to be insider observers? Any tips for how to practice these?

    ReplyDelete
  2. Pam,
    I'm so glad that you brought up the challenge of being an observer and staying out of the learning. I agree that it's challenging. We recently had a staff member observing judges at the State Fair, and she came back from an excellent observation and said that it was so challenging to not join in the conversation because it was fun! I think some types of observation can be very flexible and so interacting (to keep from losing a "teachable moment") isn't going to ruin the project. For example, when I worked with early childhood teachers, we would observe where students were playing 2 to 3 times throughout the day to get a log of activity in the classroom. The goal was to get this accomplished within a couple of minutes, but sometimes a child needed your attention, so a teacher would have to go back to it later. Having this flexibly is what I think is great about self-assessments that are designed by practitioner that don't have strict research purposes.
    I'm interested to hear others reactions to the merit of stepping into an observation or staying completely removed. What works for you?
    Thanks,
    Sam

    ReplyDelete
  3. Hi Sam,
    I, as you know, am also a supporter of using observations after school to assess quality of programming. When program staff are reflective about point-of-service practices, program improvements can be made!
    I also agree that it is certainly challenging to be an 'inside observer,' as Pam mentioned. Even as an external evaluator, having students ask me questions or request additional assistance has occurred from time to time. Generally, I redirect the students to the program staff (as those are the individuals for which the students need to establish relationships, feel that they can ask questions, and rely on throughout the remainder of the program). Additionally, I want to see how the program staff respond, as that is one key part of the observation!
    However, from time to time, I discover that program staff are too busy or are unable to meet the needs of all students. When this occurs, I do respond to student questions/needs - as the students need it and I definitely want to lend a hand. However, that experience, in and of itself, lends information regarding key areas that the program might need to be developed. Perhaps program staff need additional training on managing the needs of students - or perhaps additional staff need to be hired so that the student to staff ratio is lower.
    Although remaining objective and external to the situation is certainly a challenge to conducting observations and self-assessments, I image an equal challenge exists in providing feedback/observation results to colleagues. As an external evaluator, my role is to provide a critical eye. However, I'm interested to hear practitioners experience with this. Is it challenging to remain true to the results and tell it like it is?
    Jenell Holstead
    University of Wisconsin-Green Bay

    ReplyDelete
  4. Hi Jenell,
    I appreciate having you join in the discussion. You raise a good point about having interaction with youth during an observation as being something worth noting. Part of being an observer is noticing all of those details and then adding it to your notes and potentially to your conversation with staff.
    I think your point about the challenges inherent in being either an external or internal reviewer is important. I have always found more comfort in observing in an internal role, but I blame that on having the luxury of starting my observation career doing observations in an external role with the benefit of a unobtrusive screening room. I didn't have to worry about people approaching me because for the most part, they couldn't tell that I was there!
    What do others think about being an external or internal reviewer? What do you find challenging about either role?
    Thanks,
    Samantha

    ReplyDelete
  5. This discussion reminds me of my first observation using the YPQA tool as part of the MN 4-H Program Quality Improvement pilot. I was observing at a 4-H club in the county I used to work, so I knew many of the 4-H'ers and their families. At the beginning of the meeting they introduced me along with my co-observer, when we explained why we were there. Later, in an effort to be welcoming, they invited my co-observer and I to join them in recreation. Normally, I would have jumped right in, but needed to reiterate that our role was just to observe. I've found it much easier to observe in other settings where I do not know any (or at least not many) of the participants.
    I also think you bring up a good point about not only using a standardized tool, but also the need for creating a site-specific tool as well as developing skills for "anytime" observation. Building skills in staff to be able to adapt observation for program improvement is a worthy goal. What skills should we focus on as we look to build our competency in observation?

    ReplyDelete
  6. Hi Becky,
    Thanks for sharing your experience. It is often easier to watch a group that you don't interact with regularly. I do wonder if you notice more or less in these environments.
    To answer your question, I would say that there is a definite art to observation, and from my experience the best way to get comfortable is to jump in and start practicing. The more that you conduct observations, the better you become. I also think there is some merit to starting with a more structured observing (maybe like the YPQA) so you have something to guide you. Then as you become more comfortable, you may be able to modify observations to suit your needs.
    How about others...what skills do we need to build observation competencies?
    Sam

    ReplyDelete
  7. I'm glad to see that leaders at this event are more interested in becoming reflective on their students and not come across as a know-all leader. Great perspective on hearing about a youth program gone right with good "centergy" in helping kids out. Can't help but wonder how the food was though, (guy thing)..., were there hamburgers and hot dogs, or just bottled water and cold sandwiches? Good food can help bring down stress levels for both educators at this program as well as kids.

    ReplyDelete

Post a Comment