For April 27th class:
The 2-3 Analogous Situations we explored and what we learned:
1) Google Employee Performance Reviews
- Transparency — Transparency in performance reviews and expectations varies a lot across business units. To address this, Jeanne’s group developed a “Skills matrix” that laid out the behaviors expected at each level for each performance level. The matrix was then used in reviews to explain performance ratings, and also as a guideline for developing an action plan for each employee, based on which areas in the matrix received lower ratings. One criticism is that the skills matrix was still fairly generic (e.g., consistent communication vs.
- Static view and forward looking — Employees received bi-annually formal feedback from managers that rated their current performance along several dimensions. When they met with managers, they also developed an “Action Plan” for how to get to the next level.
- Objective and subjective feedback combined — Managers rated employees on certain dimensions on a 1.0-5.0 scale. Note – they did not communicate the numbers in reviews, but instead had phrases that the #s translated too — e.g., 3.0 = “Meets expectations”. Also, verbal feedback on strengths and areas to work on was gathered from 3-5 people to get a 360-degree perspective. The employee proposed the people to write these reviews; the manager reviewed the list and changed it as they saw fit; some groups shared the direct quotes of feedback and in other areas managers consolidated and paraphrased the feedback, into one coherent review that anonymized the reviewers.
- Exponentially tied to rewards — Performance reviews provide exponential upside. E.g., 3.0 means you get 100% of target bonus, 3.2 means 104%, 3.8 means 134%
- Fit to a curve — Google aims to have an average review score of 3.4
2) Zagat Restaurant Rating Process and Data Management
Duel-pronged rating system:
- Users generate "ratings" by giving restaurants a numerical score from 0-3 along different dimensions (food, service, etc.) - this score is converted into the ratings we see by an advanced computer algorithm
- Zagat staff (including statisticians) keep records of past reviews and check factual accuracy of new and old reviews (i.e. they know the ins and outs of the data for their respective geographies)
Potential translation for our project: Teachers as "users" who collect test/other current student data and schools as "Zagat staff" who maintain more factual/demographic/past performance data on students
Data usage and management:
- Data received through online customer surveys and subsequently compiled into the paper and online guides we associate with Zagat
- Deeper use for data along demographic lines/trends is sold through B2B channel (e.g., First Republic Bank wants to know what restaurants resonate with single women in mid-30s with income of $60K or higher to use to target clients)
Potential translation for our project: Need to understand not just higher level trends but individual student trends to target areas of strength and relative weakness
- Surveys are time consuming and after 8 screens of questions, 70% of people abandon the survey
- Subscribers to Zagat are representative of who Zagat used to target but not who it wants to target (i.e. they need to broaden base of raters to represent new clientele)
Potential translation for our project: 1) Time is a huge issue for teachers 2) Our product can't just self-select teachers who are comfortable with technology but must have mass appeal in order to be effective
3) Chevron Oil Refinery
A) Assessment Overview:
Chevron uses a very complex, quantitative system to measure physical and
chemical elements within their refineries. Tens of thousands of electronic and
mechanical sensors are constantly reporting information of the on-goings within
the refinery. The engineers have developed complex computer algorithms to
analyze all of the inputs. Assessment is
essential for ensuring that the plant runs efficiently, profitably and
safely. There are too many sensors
reporting data for humans alone to analyze the constant stream of
information. The engineers must rely on
computers to process the data and report relevant findings to the engineers so
that they can manage the refinery appropriately. Beyond the traditional monitoring of the
current conditions and operations of the plant, the computers use complex
algorithms to analyze the historical trends to both highlight red flags and
required interventions as well as predictive
B) Historical Analysis - Red Flags &
From an historical standpoint, Chevron is able to tell which components or
even systems within the refinery have been causing multiple problems and may be
in need of an intervention or overhaul. From
a classroom perspective, a system of like this may be able to warn teachers
that a student has been delinquent in multiple classes so that they can take
steps to intervene in the problem, rather than waiting for one teacher to
observe multiple instances of delinquency before intervening.
C) Predictive Assessments:
Understanding how components and stems function in relation
to one another helps engineers predict where issues may emerge so they can take
preventative measures. From a classroom
perspective, teachers could be warned that because kids struggled with concepts
X and Y, it is likely that they will struggle with concept Z (but not
necessarily concept A), so the teacher should spend more time reviewing concept
Feedback on our rough prototype:
What we learned about the user from our rough prototype:
The user wants feedback and transparency. E.g., Assessment tool should help teacher understand where students are AND it could be used to provide feedback and transparency to students for where they are.
Our most current POV statement (descrip of user) needs ______ (think verb) b/c ______ (insight). :
[INSERT TEACHER NAME HERE] is a data-starved, moderately tech-savvy, enthusiastic teacher who wants an easy and engaging way to increase the breadth and access of real-time and cumulative assessment data for his/her students so that he/she can better tailor curriculum to match student needs, identify struggling students and move through material at the appropriate pace.
1) Technology and Time hurdles prevent teachers from using this concept
2) Need to use technology to quickly distill key insights for teachers
3) Need to facilitate the collection of data to reduce burden on Teachers
4) Coordinating students to input assessment data is a complex process
5) Student assessment data can be uses to track teacher effectiveness
Areas for Help:
1) How would you use this in your classroom? (Pre-tests? Interim Assessments? Private vs. Public results? Bank of Samples vs. Customizable?) What are the limitations for you using this more?
2) What level of time/technology hurdle is acceptable?
3) What are you biggest frustrations or roadblocks in doing more assessments?
ideas/components/aspects we are going to prototype on Thursday, April 29th:
- Rubric/Skills matrix to structure multiple aspects of assessment -- And how does assessment tie to curriculum?
- Using assessment to group students -- in common ability level? matching up students that have mastered material with students that need help?
“We take a couple big assessments per year, but we don’t get
the results until the year is over. The
information is too old to really be useful and it comes in an illegible format –
We just leave the CDs on a shelf to collect dust.”
Green Dot Administrator
“I used to spend my lunch break grading quizzes from the
morning so that I could group students more effectively – Most teachers would never
think of making the time for more immediate assessments.”
Teach for AmericaTeacher
Map of insights for teacher tracking device
Business Plan class exercise
Customers: *Teachers (end users)*, Schools (principals, CTOs, etc.), Districts, Foundations, Parents (PTA), DOE
Key risks are ...
Desirability - Teacher pain point might not be big enough;
Feasibility - Creating a tool that meets 'holistic' child assessment approach
Viability -Will data prove useful? Is real-time input practical?
To be successful we must ...
D - Convince customers that data on 'holistic' child is critical
F - Hire talented programmers; Build connections with foundations and districts
V - Application must be super intuitive, user-friendly; data analysis should be automated and useful
Next steps are ...
D -Determine size and scope of 'pain points'
F -Determine market for programmers; Approach foundations
V -Build prototype and test
$500K over 6 months
HMW enable teachers to easily gather more holistic and real time data on student behavior and engagement with minimal burden
Mr/Mrs Teacher is a data-starved, time-constrained, enthusiastic teacher who needs an easy, real-time way to capture a more holistic view of student performance (engagement and behavior).