Event ID: 2860668 Event Started: 3/16/2016 7:00:00 PM ---------- Please stand by for realtime captions. >> I am showing we are at the top of the hour. This is Robbin Bull with NCDB and I want to begin by welcoming everybody. I'm going to go through some housekeeping items before hand it over to Linda McDowell who is going to be kicking off today's webinar. As you have notice, all phone lines have been muted so we don't have to worry about background noise. The question-and-answer session will occur at the end of the presentation. However, you can write questions in the chat box at any time during the presentation. As it will be monitored throughout the webinar. We want you to know that this webinar will be recorded and archived for future viewing. I am going to start the recording now. You will hear an announcement and Linda, that will be your cue to start. >> It's a happy time to be here with you all. I am looking at the list of people and our participant list and most of you I know. For those of you I don't know, my name is Linda McDowell. And I've been at NCDB for a little over a year as the Executive Director. Before moving to Oregon, I was living and working in Mississippi and I was the administrative director of the state deaf/blind project there for 16 years prior to coming to NCDB. I am confident as always that the solutions for the children and every state come from our strength as a network. State TA projects working effectively together with the assistance of the national center. The technology that you will hear about today is the result of a large amount of network activity. I do know that network life was encourage years ago by a great leader in our field, Mike Collins. And network life is OSEP current directive to us and as I said a moment ago, I am convinced that many of our technical assistance network solutions will come with the use of technology. We are a low incidence field, which means we are spread thin across the nation. In the agreement that we need can often be assisted by technology. It is a means for communicating with each other, communicating with others, holding to common standards for what children who are deaf/blind need and advocating for those common standards. You will be hearing a story today about just such a technology tool, an electronic portfolio that was developed collaboratively from our network. As you can see some of the bullets on the screen slide, you'll hear a lot about technology as you hear a description about the development of the e-portfolio. I am confident that we have agreement across our network of technical assistance projects, families of children who are deaf/blind and personal prep programs that progress for children will always be greater if the service providers around the children both the teachers and the innovators have knowledge and skills for the strategies that work best for children who are deaf/blind. Thankfully advocates at the national and the state and the stool -- school district level are hard at work to achieve the necessary recognition of the need for teachers and intervenors who possess the knowledge and skills to work effectively with children who are deaf/blind. It is a great celebration that our field has nationally recognized knowledge and skill competencies that have been agreed upon and adopted within the special education standards of Council for exceptional children. This agreement on the set of knowledge and skill competencies across our network is indeed a strengthstrength. There are differences across our states in how the training of personnel including intervenors can and should be accomplished, but we have agreements across our network that knowledge -- knowledgeable and skilled intervenors are needed and what those competencies are. Today in this webinar, we are focusing on the development of a tool that can be used by practicing intervenors. It is technology, yet another strength of our network by which intervenors can demonstrate that they possess the knowledge and skill competencies adopted by CEC. Thank you for joining us today, Amy Parker and Leanne Cook will be giving you the story of the development of this technology tool. It has been collaborative development that is been underway for over a year. In partnership with intervenors, family leaders, ATA projects and personal prep program personnel. What has been learned at each of the phase of the development to inform yet the next phase, you will hear and we will end today's presentation with an explanation of what is next. I'm hoping if you have questions along the way as the story is told that you will type them into the chat box. We will try to use the remainder of our time to answer those questions and as always, if we run out of time today, please feel free as always to contact us. At another time. Amy? >> Thank you so much Linda. Can everybody hear me okay? I am on an iPhone today. So hopefully using another form of technology to cross the distance. When Linda referred to Mike Collins in 1992, I think of Mike Collins as a Northstar for us. As a field. He was talking about modules and network and relationships and standards and identity. Identity is going to be a theme of our presentation today. Identifying roles. We are honored to have many intervenors who participated with us. We call them intervener leaders because that's what they really are. On the call with us today. Who helped us Coke create this process. But in 1992, what we didn't have to the degree that we have now is technology. We are going to be talking about technology not as a tool that replaces our network but as something that really empowers our network and empowers this identity as Linda was saying. What is a e-portfolio? I found this graphic interesting and helpful as a way to collect an individual to collect evidence. Things that show that they know what they are talking about. This symbol that is on the screen also implies multimedia. Which is another strength of a e-portfolio in this new world of literacy and communication. I don't know if you're like me and you like to get on Facebook or you like to look at YouTube but there is a whole new way to communicate with gifts and pictures and video that is just really -- it has changed the way we communicate. Almost about everything and so we are going to talk about what an e-portfolio does for identity and also for demonstrating standards. With this tool. LeeAnn, MI coming through okay on the sound? Okay. A little bit insecure about that because I'm working from home. Thank you. The purpose of an e-portfolio, one of the things that is exciting. If you were to get off of the line and just Google an e-portfolio related to practice, you would literally find thousands and thousands of hits. People have been at this since the 1990s, really when the Internet started to come online and we started to think about digital ways of connecting, which is a huge empowerment tool for low incidence field. But an e-portfolio can look really differently. This slide is talking about a gal with her PhD, Helen Barrett who said in describing an e-portfolio, these are digital stories that represent deploring. They can be categorized into these different groups, at their most basic level, an e-portfolio could look like a scrapbook. Work could look like a [ Indiscernible ] or a resume. It could be just a collaborative tool. I say just a collaborative tool because that is also really an important thing for our field to have collaborative tools to look at something together and talk about it. But there is this last level here. This authentic authoritative evidence and our purpose and designing this you portfolio, that is pushing this whole conversation in a very specific direction. As Linda talked about, rational standards are also a Northstar for us. It is a guidepost for us for our we on the right track? It is also something that in special education, there is a great and glorious benefit to being a part of something larger than your self. To say you are a part of this field. In this field is special education. And when you talk the language of standards, that is something that states and administrators and other people even that don't understand deaf/blindness can understand. That you are following a standard. You have standards for your field. This CEC, the Council for exceptional children is the organization or the body that helps develop standards. There anybody that develop standards in special education. They are highly respected. They are internationally known. They are respected by OSEP. But they have all of these different categories of standards. But for us what is fortunate is that there are standards for intervenors that have been through a process and have been validated and there are standards for teachers as well. So standards are seen as a maturation, a marker of maturation in the field. When we implement standards, that's when it becomes real. Not just us, not just our quirky little field of deaf/blindness, this is something recognizable that we can talk about and that other people can understand these standards for excellence as well. So it an e-portfolio, what I think is interesting here, other types of portfolios exist, but e-portfolios have been known to help us link and think about competencies. In a way that makes it and identity. This identity conversation. Needed. What intervenors Toles in 2012 is that they didn't have an identity. Most able don't understand what they do. That is a strategy, e-portfolios are a strategy for helping change that. So this is a lovely picture of the eight intervenors who dove into this process with us. There were four states -- for state deaf/blind projects that were represented in this project in designing the beta test and there were a couple of university partners who also helped us think about standards and implementation. And we are going to talk about designing a path that is both sustainable and scalable. We needed a way to cross the distance, to time travel in some ways to create a way to create shared expertise and to have some more competency-based dialogue. >> I'm not going to belabor this too long because I want to get into what LeeAnn is going to share with you in a few moments. We do want to point out a couple of parameters for this work that we engaged in. Our e-portfolio design was designed to assess an individual's competence. This was not based on a person's program of study. So it wasn't a study of the person's training program. We were trying to set up a measure that was based on the national standards that people could say yes, this person has had training. But what do they know? And what can they do? And what can they demonstrate? These e-portfolio dialogues, one thing that was exciting that happened was that it did fill -- still allow people who engaged in training intervenors to have dialogue with each other. About how they did training. That was in the intent of the e-portfolio, the intent of the e-portfolio was to have individuals demonstrate competence. But it also provided us with a dialogue across training programs to talk about how do you embed the competency into your training? How do you talk about them, how do you recognize them, what assignments do have that align with them? Dialogues in other professions like nursing and in medical school have said e-portfolios are critical for developing interprofessional collaboration. And even in our own little study we saw that. We saw that happen. Interprofessional collaboration in this process. Practicing intervenors were included in this beta test who had recognized roles and who could receive some support. And mentoring advice from states. nursing, in teaching, in medical practices, where people use e-portfolios, and the reason why we designed it as an e-portfolio to align with the literature is we want this process to have validity. We wanted it to be authentic. And as I said at the end, validity also means where if valuators can look at this and recognize this practice together. The whole goal is that and what parents have said to us again and again, we know that this exists in other states. We wanted to exist in our state. We wanted to exist in every state. OSEP ones that as well. We have to design a process that is valid that says this is aligned with national standards that can be recognized and can use those new literacy and media that we talked about. Very quickly, if -- these are just more references for you that portfolios have been seen as a way to bring together complex assessments and so we designed this beta test with intervenors. Again, alignment with the CCC -- CEC competencies allowed us to have another type of validity. That the CEC competencies are approved through a validation process. First through a white paper, and then through systematic surveying, then to review and they are aligned with standards in the CEC that are across all the special education areas and we just went through this again. We've gone down know very recently our beta test was designed around 10 standards because that's what it was when we started. And now the CEC has realigned all of there competency bases including for teachers of the deaf/blind and intervenors around seven standards. Unfortunately for us it didn't change at the competency level but it did align everything at the standard level. We will get more into that later.LeeAnn is going to show you. Another really important piece as you look at this beautiful image of this little child exploring a fountain with her cane and I believe that is her mother or teacher with her, I don't believe it is an intervener. We get these beautiful videos and images from all of our state partners to really increase the visibility of children who are deaf/blind nationally. But when you are talking about reliability, and this is also in the literature, you need to have an instrument or tool like an e-portfolio where you can have two people that understand intervention look at it, use a rubric or scoring protocol and come up with a similar view of what they see. Based on the standard. So that it is not the world according to Amy, the world according to Linda or LeeAnn, that it really is based on the standards and it is based on the practice. So we began with our internal team to develop protocols and to use them to look at the intervenors, the intervenors who created their portfolios and open them up to us. To have open conversations about that. And then we've gotten into this last phase where we just recently had portfolios reviewed by external reviewers. And that was a real test of trust as well. To launch those portfolios out there to teach new people to use a protocol and to look at those scoresscores. Now I will turn it over to LeeAnn to talk about the technology platforms that we used and that we will use in the future based on all this good data. LeeAnn? >> You are still on mute LeeAnn. Still can't hear you. >> We started this process we found a platform and it was called [ Indiscernible ]. It was accepted internationally as a portfolio platform. It was built around [ Indiscernible ] and being able to use a variation of artifacts we will talk about in a little bit to really demonstrate knowledge and skill. The system is very robust and we had to tailor it interim some edges, give it a little haircut. But then we started to outgrow the system that we tailored that we outgrew it in ways that the system wasn't intentionally built to do and the main area around this is -- a run-scoring. [ Indiscernible ] was not intended but it doesn't successfully work with the kind of scoring that we are hoping to do and doing with these e-portfolios. We decided to make the decision to build a platform that suits the needs that we have and that the community has for this process. And we will talk more about data and those things, but one of the things has been -- and you will see throughout all the data one of the most paramount points is the transparency of all the decisions we have made are founded in evidence not only in patterns that we have seen but narrative evidence for people who are participating in it. When we say we have made a decision, it is a collective decision based in evidence, not something that we just think would be the best. But that we can ground and concretely say this is the right path for us to go down because it's a little bit -- it would be us taking too much liberty to say we think this is the best so we are going to do it. But we would like to move collectively towards this new certification. We work -- we are going to talk about an artifact and what that means. There are no eight kinds of artifacts. People say what does might -- what is my portfolio made up? Multimedia content. It can be coursework, if you have been involved in courses, videos if you are working with a student or an adult, it could be a performance evaluation, you are talking about that, and artifacts are more than just a documentation. It also includes you talking about what that documentation is and what that says about your practice. But we most often get asked what do I make my portfolio of? What is his content and how do I show you that I know and am practicing effective skills of intervention? And if you look at number nine on this list, letters of reference or recommendation those were originally included as an artifact type but we found that they survey better for first in the portfolio in another location. For so -- more formalized saying that someone is in support or more of a letter of support saying that the person is a great intervener and they stand behind them. Continuing on, the artifact does have the documentation which is the physical -- this is a work sample. Performance evaluation. But then there is an explanation. I don't know if Amy you would like to talk about this at all. >> I think you are doing well. Do you want me to take this one as far as in the literature I know that you could just geek out with me on the literature. Anyone is fascinated with the literature as I am. These are called reflections. This is a beautiful image of one of our intervener leaders working with the student that she served. And in this image, if you envision and a portfolio, let's envision that she was using this image to talk about her practice and what she is doing. She would describe this photograph perhaps as an evidence of something that she is doing. She might provide some more information about it going back to that artifact list. But then she would explain how -- what she is doing in this image or other artifact, really demonstrates her practice. So this was part of the great work that our intervener leader stepped up and did with us. These eight women -- think about how complex this is. If you think about your own life and what you do, you just do it. We do these things. But then when you have to explain it to someone else, you have to take it out of your self and that is what and a portfolio does. Out of yourself, out of your practice. Demonstrated and then you reflect on it. And do it. This is very big in the literature. That is what I wanted to say about that. Anything else LeeAnn or are you going to take it not? >> One of the things we notice with our intervenors one of the most delightful areas of growth is that we saw them through explanation learn about reflective practice. And the importance of looking back and being able to talk about their work. And seeing the growth of their changes or things that they could do to improve or are things that they were proud of in their professional life. And I was something that I don't think oftentimes our intervenors got to see and got to see growth not only in their students but in themselves professionally and that was the explanations give our intervenors a lot of confidence in their practices will be able to talk about what their work is and how they do it. So I know we have thrown a lot of words at you. So I'm going to show you an image that can help you see how all these bits fit together. There is a very hierarchical structure to the eat portfolio -- e-portfolio. All 79 of them and their clustered together in groups of five or less to build an artifact. And artifact is the documentation and the explanation of why this is evidence of their practice. And then those artifacts build a standard which is one of the seven standards the CDC has set forth. And then your portfolio is a collection of those standards. This is a hierarchical structure from the smallest unit being the competency to the large portfolio and how they all fit together. I would like to show you a little bit of what this actually looks like so you can see what these things manifest us. This is an artifact, a complete artifact. This one has been reviewed. This is only part of this documentation. But at the top under the title data collection it says which competencies that unit is the smallest unit and then underneath there is a rainbow colored piece of paper that says learning zone and this is one piece of evidence that this intervener created to work with their student. This is what they did in their practice to show that they have the knowledge of how to do these competencies. And then on the right-hand side you will see some questions and say format. What is that? Where did it take place and then the explanation guided by prompts. We expect these intervenors not to be able to just talk about what this artifacts is and does but to respond to questions about what is this about your practice. How does it address these? Here's a single artifact and if we look at a standard, this is to artifacts on a single page. This is in our old systems what our new system it's a little bit cleaner. A little more beautiful. Easy on the eyes. But this is a collection of artifact. And then one step higher is the portfolio. And this is when I was 10 standards and an about me section when the intervener had an opportunity to talk about their practice, their work practice and about their students. Amy? >> LeeAnn, I wanted to take one minute if you could go back to slides. To the slide where there was the learning artifact. These prompts that LeeAnn is showing you, it is in a small font here. Just hold the hook here in your mind. She will come back to this and talk about it later. These prompts were very important because they were codesigned by the intervener leaders with input from the state mentors and our University. This is how we helped create that reflective practice that LeeAnn was talking about. It was very hard for people to even explain some of what they were doing adequately for us for an independent reviewer. Not because they didn't know it because that's a pretty hard skill to learn. And they were doing many things at once. Collecting artifacts, putting them into a new e-portfolio system, and then talking about them. And many times our partner said we love you guys, but we are on cognitive overload. This is a lot to take in. We are learning this whole new Mahara system. We are learning how to get our artifacts together, we are storing them on our hard drives and uploading them. And then we have to explain to someone else what they are. So these prompts here, these questions, that was a part of our code design process and a part of what LeeAnn was talking about with reflective practice where that explanation, that explanation part. Go ahead LeeAnn and I know you will come back to that later. >> Skipping ahead I wanted to quickly show you a little both -- bit of our review process. Those of you on the line got the opportunity to review. The review processes rather automated. We have the scores based on collective data around rubrics. It is as simple as plug and showed in the most simple fashion. You select the level of competence around the documentation. The physical piece of evidence that they use whether it be a video or a document they wrote or an image and the counter system they created and then lastly the evaluation of their explanation and an area for to leave comments about their work. I would like to talk a little bit more about data sources and senses process was so data-driven because I think at the core of this the value of transparency is one of our highest values truly is that this isn't a process that NCDB is created, this is a process that we as a community are creating. Because it is to be sustained. And to live on for children to benefit from. We used surveys. We had focus groups and one of the greatest sources of data that we found especially technologically is what we call the help desk. In the moments of panic and frustration that all of us felt and NCDB and of course our intervenors, they would simply type out what a problem was and we would respond to them and go through this negotiation of what is happening and how can we fix this. And we had peer dialogues and also the internal and external scoring of the types of artifacts people use and all of these pieces of data came together in this nexus of creation of a new system. We brought all these pieces and we would specifically go back and find comments where someone would say I would really love it if we could do thisthis. Or I am so frustrated, can you make this happen? Or how do I talk about this part of my work. And we use those points of data to really drive forward what we are doing in the portfolio. Sorry I didn't click next but you can see that information out and how this was really a pivotal part of our process. And being a collective group, and also creating a community around this whole process. Which is something that the intervenors really initiated and they really drove a lot of these conversations about what do I do next? How do I do this? This is too much, this is too little. And they help us drive a lot. I will pass this to Amy so she can talk about the different sub phases we went through per se. As we went through the train tracks of this process. >> I think you can see why LeeAnn was so integral to this process because she loves data. And she was really -- she and Jenna and Jeff and our team and NCDB were also very much in this together. With all of those people in the rainbow Circle that she showed you. With the intervenors. Really wanting this to succeed. This is been a fast train. My son likes trains. He is on the autism spectrum and he love strains so we had trained dialogues a lot in our family. But if you think about the two tracks in your mind of what people were doing, we called this the industrial phase. Because what we needed to do was launch the Mahara system and be able just to get in there and use something. We couldn't spend all of our time designing something perfect, so we grabbed something that was free. And we said we are going to use this to just get in and learn the technology. And kick the tires on it. It's not perfect, we know that, we warn people when they came in, you have to have a high tolerance for the technology frustrations that you will face. Get ready, you are going to get frustrated. And people had a good sense of humor overall about that. They used to be help desk and they said we are going to kick the tires. And so at the same time, that is one track. The technology track. Learning Mahara.At the same time the other track was learning about your practice. Owing by the way, we are going to talk about 126 competencies. No pressure. We can't sing the praises enough of our intervenors and our state partners who said okay, we are going to figure this out together. And broke down and started looking at this is what my practice means. We are going to talk about it. This was the industrial phase. If you think about what an industrial phase looks like, it is a lot of building. It is like a construction site at the very beginning. It is rough. We had to talk about from your practice, what you are ready have, your documents, your certificates, getting all of those things organized into a folder, start talking with each other, that was another beautiful thing that happened. The industrial phase. Okay Amy, that's enough of that. Because we are sometimes overly ambitious with our timetables, we did review. This is an image of our scoring rubric where yes, we evaluated all 126 competencies were scored when people submitted them, and it was -- each competency was ranked. So at first, our interrater reliability an agreement was also very low. But mathematically you would expect that because the more you have to agree on, the lower your agreement will actually be. And look at this, we divided it into advanced, proficient, emerging and no evidence. And had some descriptions across those four things so a disagreement meant if someone checked advanced, and someone else checked proficient, that was a disagreement. What did we learn from our industrial phase? We learned a lot. We learned all things about not doing it only at the competency-based level. Artifacts themselves lend themselves to chunks of competencies all around them. So in a video or even in a well described picture, you can link several competencies. LeeAnn, I will let you take over here. >> In this phase, it was very again, industrial was such a good description because we were really looking -- a lot of the things we saw on this phase were technological because those were things that clouded us. And they were distractors to what this really was. This was one of the biggest phases for technological -- more the industrial revolution.We realized what we wanted and didn't want.And then we started slowly implementing those changes. And again the data here was so driven by the intervenors. And whether they knew it or not, we were tallying their numbers and saying these are the things we need to fix and change. So this stage although we realized again from our rubrics and data collected Rhonda rubrics that it was a little ridiculous to try to create 126 competencies individually but we needed to start to reevaluate technologically what can we accomplish here. >> LeeAnn did you want me to do this refinement stage? I will go into it really quickly. We are going to have to speed the train out, staying with our train metaphor. We learned a lot, this this whole slide. We learned a lot from the intervenors and we talked a lot about that the most important piece of the artifact, of the e-portfolio was what the demonstration of competence. The e-portfolio did not need to become a writing test. We had lots of significant conversation there. The artifact became the unit of analysis. What that means is it was a video or described picture or work sample or observation, and intervener would select competencies that matched with that. And that artifact would be the thing that became reviewed. One of the most cool things that happened out of this phase, we met in Utah, many of you saw the intervenors present. They had a celebrate -- celebratory moment together in completing this draft space. That's with the literature says within e-portfolio there is lots of celebration. They not only don't with all of this industrial phase, they had a celebration at the end of it. They formed some relationships. They set up a private Facebook page together. And really we can't say enough that the deeper connections that were formed, the intervenors supported each other as colleagues and the mentors were vital in this process. In supporting the revisions. This is the refinement stage that we went through through until early November. We tried another rubric. This is a sample of our other rubric. We can send it to you but it did not survive. Know that this top score, this artifact element was confusing and it wasn't as relevant to the scoring process. We decided to move on and really stick with the documentation. Does the artifacts show the competency. Can someone see them and how do they explain and we talked about waiting, the weight or scoring on the documentation being higher than the explanation. So what we did at this phase from September to November, a lot of work was done. And some people quite naturally couldn't continue with us. This is the other thing that happens. People get pregnant, people have other commitments in their lives and it is a big deal to do and a portfolio. People move, this is what happens in education. So we also had some intervenors who were with us in phase one that couldn't continue with us for very natural reasons. Out of this we developed an about me section. We moved letters of reference to the about me section. LeeAnn, I don't know if you would like to talk about the about me and some of the other things on this slide. The writing prompts which I mentioned before. >> Very quickly, the about me again are intervenors. They were wonderful. A few of you are on the call but you deserve much more praise than you have gotten from us. They got together of their own free well, we did not ask them, came to a meeting and said we have some prompts for you because we think the about me needs to be clear. And we sat there and said we did the same thing. And then all of a sudden we had the exact same prompt questions, build from both sides. So it was a beautiful moment of our train was in the same spot at the same time and we didn't even plan for. Our intervenors design these questions so you could get to know them and their practice. In a single page. A single read of saying this is where I am, this is what I do and this is where work with. And the other thing -- our intervenors did is they collectively as a group in groups wrote writing prompts for one we had different types of artifact. And then from surveys and synthesis since we have gotten rid of artifact type because that was cognitively a little bit too much on our part and for the intervenors to try to differentiate all these different kinds of artifact, and what they are, we then synthesized writing prompts that would go across the entire portfolio see could always expect to see these questions and to answer them. Along with that we were finally checkboxes which is background information saying what is this piece of documentation? I don't know what that is but as a reviewer you need to know. Very quickly we have been looking at it. And along the same vein of these what is happening, we need to change this. With the removal of artifact type we realized because intervenors were already doing this in their portfolios put that we saw the pattern that they were mixing multiple pictures with video, or a performance evaluation with a video of them showing that they not only have the knowledge as documented by someone else but I can show you that I have this knowledge and know how to use it. So they started mixing different kinds of artifact types. Videos and pictures and letters of reference with performance evaluations with documents that they had made for their students. So now you could mix and Max -- mix and match like a buffet and that was one of the things that are intervenors drove without them even knowing they did it. Back to you Amy. >> We have a question in the chat pod from Tina and it's a great one. It did come up for us. Tina rates how does the intervener separate the work that they do to create the artifact from the work that is done by the students team as a whole? For example, the intervener may take suggestions regarding a specific calendar system but the team discusses and designs the system. It's a beautiful question because we really did talk about the intervenors role as the implementer of the team. So that played out with -- let me give you an example. One of our intervenors worked with a child that had some really involved medical needs. And she was very clear in the presentation of her work when she was positioning the child or when she was moving the child into certain positions using specific strategies that that work was reflective of her process with the OT, the PT, the team. The teacher. And that she was implementing that plan in that routine. So it's a beautiful question Tina and it did come upup. And it was also one of those things that it intervener did have to talk about. What is my practice? What do I do? Versus what is the design of a team? And that also came up in our review as well. So when a reviewer was looking at something, they also wanted to see the intervener talk about how they were implementing that plan. As a member, a vital member of the team that provided continuity for the child. But also that the intervener was not responsible alone for designing the calendar system or the intervention. So we had multiple examples of that play out but it is a beautiful question Tina, and it became woven into the design. Also, the about me section helps with that. To get back to the slides very quickly, the about me section was meant to help contextualize the intervenors practice. So again, who are they? How are they trained? How was I trained as an intervener? Where did I get my training from? Who am I serving? We talked a lot about the diversity of deaf/blind people who are served by intervenors. So again I mentioned an intervener that was serving a child with a lot of complex medical and physical needs versus another intervener that was serving a child that was on an academic track, using advanced sign language, maybe participating more fully in a regular curriculum. So those kinds of conversations in about me help contextualize the intervenors practice as who she or he is serving in those contexts. And we again, the intervenors knew that the about me section that they codesigned was not going to be scored, but it was going to be used as an anchor. We need to move a little more quickly. Basically through this portion we came up with an agreement across our whole team to have documentation be weighted 60% an explanation again to be weighted 40%. We also spent a lot of time talking about intervenors whose first language may not be English. Maybe having alternate means to present an explanation. Maybe and American Sign Language, maybe through talking, maybe through a video interview. Again, this portfolio is not a writing test. We really spent some time talking about that and how the explanation is meant to support and align the documentation. Not take over the documentation. So I will turn it back to LeeAnn. This is the most involved slide and I know we are quickly running out of time. We had so much data to share with you. We do want to leave a few minutes for questions. LeeAnn? >> On this slide we have our intervener along the far left hand side with names given to them and on our St. Patrick's Day coming up we always do things with themes around here. And our first phase we had initial reviewers, this is that pale yellow column, the most left column with internal reviewers who were from the state.And they at the competency level scored each competency. And of course if you have 126 possible competencies that you can give for different scores for you will not get very high results. We had them ranging from 25 to 66%. Which is quite good. But still not in the range we were looking for and we documented in our research we were looking for 80% interobserver agreement which is the standard of agreement to be acceptable in the process. So then in phase one part B-letter again this is when we realized the competency level is not going to work. Let's look at the artifact level. There are two columns, the one on the left shows at the competency level what their I/O a score was when every single competency was greater. The one on the right shows when they were clustered what they were. A 22% to a 61% jump. That is huge. More than doubling their scores just by clustering their competencies into and grading the competencies as clusters really gave them a lot more agreement and connecting these competencies together around a single artifact and looking at them that way really created a lot more agreement. We did have one instance where the scored to go down and that was just from a miscommunication error. No one is at fault, but it was -- that happens. And I always love when they're something like that in our data because it shows we are working hard. Things happen. Looking at phase two of the data, this is when we get to our concrete rubric and where we are right now. This is our most recent data. These reviews were completed about two weeks ago. So in scoring when you had to reviewers and there is a discrepancy between those two a third reviewer is called into settle the argument. And in all of our intervener cases we did have this happen it is okay but we are still continuing to follow her protocol saying if there is a discrepancy that is too large we need to have someone come and settled this discussion. When we had I away at the artifact level. 60%, 72%, 75%. So we are way closer than our initial 36%, 25%, so our numbers are going up which is a great site showing that we are getting much closer to where we need to be. This also shows us that with greater reviewer trading we may able to get the score away from our current rubric. What we started to realize is it's not so much about the artifact level it's about the standard level. And are these intervenors passing their standards? They may do well in one area and out as well and others because maybe their student is more focused or doesn't often use that competency. That we looked at I away at the standard level. Are they proficient? Emerging? Advanced? Where are they. And to reviewers, with 75% which is amazing. These guys almost all the time three out of four times agree on the scores that they have given. But we were shooting for 80%. That is what we stated and we will stick to it. What we called in third reviewer per our protocol we were excited that we reached above 80% on every single person. That shows us that our protocol at the standard level worked for achieving the I away. Our rubric is working and it is showing that we are getting to a point now with the scoring of these portfolios that is becoming more valid. And more trustworthy because these intervenors are having agreement among their reviewers on the scores that's they should be giving. Amy do you have anything to say on this? >> I just want to say that it's been a hard -- it's a hard path but it's been a participatory want so at every color of the rainbow that you see here, it has involved discussion with the team, involved group decision-making, and participatory design. And in the field of nursing or in medicine or in teaching, many people say that interim server agreement at 70% on a clinical practice is good. But we were going for a higher level of -- 80% agreement that means that if something is going to be sensible, if someday we all have this dream that and intervener once they get a national certificate that it also means that they get an increase in pay. That it's a respectable practice, that people recognize that. That they will know that that's one protocol has reliability and validity for the practice. Which is what we designed. So I know that we need to wrap up here. We still have a lot more to implement. With our team. We have not had a chance to come back together with our team yet an attack some of this data. But we are looking at some of these refinement pieces even still from this last phase of design. We are going to look at the evaluation of the scoring weights. As I mentioned, we have gone down from 10 to 7 standards with the CEC. From 79 competencies. That means there will be reorganization of the e-portfolio system. Refinement of writing prompts. We will talk more with you later about what a compensatory model means. But what it basically means in my primitive brain, what it means is if somebody sees someone at a 2 and I see them at a 3 at an artifact level, on the overall standard, it is more important that you have agreement on the overall standard. Because all of those scores calculate together to a percentage per page. So one of the things we did that was a benefit of the e-portfolio system is not only did we have that auto calculation feature from that clickable form that LeeAnn showed you, we got to where it was weighted and scored at the page level. Were observer agreement was calculated and also those weights were calculated. And it is more important to see agreement at a standard level then on one isolated artifact. We know that this is a lot more involved to designing a good ecosystem, to really design a system wall. We have to think about what it costs intervenors to apply. We have to think a lot about our training protocols. What makes an effective mentor for E coaching. All things we are going to be working on with our community to design. We have to improve our reviewer training. And it's not going to be on us alone. Dr. McDowell our boss has -- is going to talk a little bit more about the next phase. Of what we are going to be doing with OSEP blessing. So I will turn it over to LeeAnn and then Linda to talk about the last slide. >> This is just a down and dirty model of what we are looking at right now with our process. And where we are headed. We have the intervener in this ecosystem that we are talking about and we have NCDB and at this point we're looking at the training and how does this all involve which we are still working with managing agency that will be in charge of this. And the actual process of certifying and how does this all -- how does this work in this bit of transition as they take on this role. We have just looking at mentors and how do they become involved and support the intervener through this process? As well is portfolio reviewers in the review board that we are looking at creating which is really exciting. Once and intervener submits our portfolio is looked at twice. They will be center a quick to the agency and notified that they need to award a decision on the portfolio. If it is low as we talked about, the third reviewer will be recruited and break the fancy what score the portfolio should be given and [ Indiscernible ] the managing agency to contact the interviewer and give a decision. This is a down and dirty model where we are at and how things are started to -- and as we evolve this model will change. And where things are headed and we wanted to give you a glimpse right now of where things are at. >> What is next? I will finish up for our time together realizing that we have some questions coming up in the chat box and I want to reiterate that we knew we potentially would run out of time. And so we ask each of you who still have questions, want more information to please contact us. As soon as you want, and we will get back to you with additional information. But what is next? All the feedback about what could be clearer within this e-portfolio platform that has been gathered via survey, the Mahara platform helped us, the ongoing dialogue with practicing intervenors, state TA project and personal partners has led to the development of an even more flexible e-portfolio Mahara platform. And in this next phase of this designing which will be this spring specific partner states who have state -based recognition of the intervener role are partnering again to have new intervenors field test this refined e-portfolio platform and they review protocol. It has been collectively named NICE, national intervener certificate e-portfolio. Very nice. Currently NCDB is working with OSEP and potential managing agency representatives to design a memorandum of understanding that will support the fall phase of implementation which is when using NICE with a national review board of professionals who are familiar with the intervener model, using a field-tested scoring protocol with fidelity, this whole operation of NICE by an agency is due to open October 1, 2016 and it will include the use of these supportive training material for intervenors, mentors, intervener training partners and reviewers in these training materials are in the use of e-portfolio. With the support from NCDB, there will be two years of study well in full operation by a managing agency that we anticipate answering then questions that current certifying and credentialing agencies are asking in regard to sustaining and e-portfolio review system. There are still questions that need to be answered. While in full operation. And in the end, participating intervener, a participating intervener in this NICE process will have met national standards to competently serve people who are deaf blind. It is exciting. I think we need to wrap it up. So as I said, we have noticed a couple of questions in the chat box. We will keep this open. Amy, did you want to try to address a couple of questions in writing or via phone? >> If people want to stay on for a few more minutes, we are certainly here to answer some questions. We know that your time is precious. Julie asked from California, are there specific training programs associated with this process? In the beginning Julie, we really did need to partner with people who had been engaged in the intervener training process which is why we worked heavily with Minnesota, Utah, Arizona and Texas. And Texas was involved with us from the standpoint of a community-based intervener. That is another thing to bring up importantly, we wanted to field test this not only was school-based intervenors but community-based intervenors as well. All of our intervenors had been through comprehensive training. That was important to start with. We couldn't start from ground zero. With a partially trained intervener. And so the states elected selected intervenors to help codesign the authenticity and validity that we are looking for. In the fall we will not have specific parameters from training programs. That being said, even at an individual level, we understand that it is going to involve and intervener being able to get some mentoring and possibly coaching and training. So that is why we use the word ecosystem as opposed to just an E portfolio a standalone process. There are other things involved. Reviewer training, mentor training and we want to work with our community to leverage those strengths of people who are invested. Deeply invested in the process of intervention. So gassed three has a question about Washington state there was an intervener training. We encourage you to work with your state deaf/blind projects to talk to them about this process. Obviously our network is vital to co-creating this process as Linda announced. Even in the launch in the fall, there will be two years, a two-year process of looking at the system, looking at how it performs, looking at how reviewers perform, looking at what materials are needed to be successful. So we know that it is not -- we just launch and then dust our hands off and then we are finished October October 1. That we have really committed ourselves to designing a good system and supporting our national network to participate in that system. You are welcome, Julie. Look forward to working with you. Other questions? >> I want to reiterate that we are available to answer questions after the webinar. In the days to come. If anybody else is reviewing the recording and wants to contact us and ask questions for verification, we welcome that. And we thank you all for coming today. And wish you all a good rest of the day. >> Thank you very much for your participation. Cindy is asking when this will be posted. This will be archived. Many people who could not participate today did want to watch the recorded. Some people who work in schools would like to watch the recorded. That will be stored in our library and I know it will be linked and if you e-mail us, we are happy to share the link. We will direct you to the link in our library. Thank you Dr. McDowell, thank you LeeAnn. Thanks everybody. Goodbye everybody. We will see you later. [ Event Concluded ]