Category Archives: Research

Bruce Baker on Value Added and Student Growth Percentiles

Posted on by
Reply

I’ve lauded Bruce Baker’s blog once already, but day by day I’m becoming an even more devoted follower of his wit and his brilliant exposés of fallacious research, bogus claims, and shallow thinking.  I’ve learned/confirmed so much by subscribing to School Finance 101, particularly about value added models and student growth percentile models, as Dr. Baker reexamines the research on these topics and highlights the shortcomings and the unintended consequences of using these measures in states all over the country.

I’m very interested in his work on student growth percentiles, first created by Damien Betebrenner at the National Center for the Improvement of Educational Assessment, www.nciea.org. First used in Colorado, and now spreading to Massachusetts, New Jersey and New York, the proponents of SGPs suggest they fix the problems inherent in value added measures.  Baker emphatically says not so, and uses the writings of the fathers of SGPs to highlight how SGPs are being misused.

To get up to speed on VAMs and SGPs and their problems as Baker sees them, go through the blog references found in his Value Added Teacher Evaluation category here, and as you read the articles, click on through to the links within each one.  You’ll get quite a wonderful look at the limitations of these statistical models as they relate to teacher evaluation.  Baker also takes on state education officials in several states, including Colorado, New Jersey and New York, with his irrevent commentary.

Of particular interest for New Yorkers is his analysis of the preliminary technical reports on the results of NY’s first SGP assessment results.  This analysis can be seen in the entry entitled “AIR Pollution in NY State…”  He offers graphs showing how factors that cannot be attributed to teachers have an effect on the patterns of SGP scores–students in low income schools generally underperform students in higher income schools, as one already knows, but the SGPs of these students also lag behind higher income schools.  This demonstrates that SGPs do not, in fact, account for effects of peer groups, of school effects, or of poverty, as some suggest.

Read Baker and weep about how assessments are regularly being misused by people who should know better,

Richard Allington on Literacy and the Common Core

Posted on by
Reply

Many teachers are concerned about the literacy implications of the Common Core.  One of the nation’s literacy experts, Dr. Richard Allington, was interviewed about the Common Core, arranged by SAANYS, the School Administrators Association of New.  SAANYS represents many supervisors, principals, and even Superintendents in NY, and of which I am a retired member.  The interview is published in the Winter 2013 issue of the SAANYS publication VANGUARD, on Special Education, which is not my expertise.  However, the interview, entitled “The Road to Literacy Instruction” is worth sharing with others in your district.  The interviewer is Peter DeWitt, an author and elementary principal in Averill Park CSD, in the Albany region of New York.  DeWitt also blogs for Education Week.  Allington is considered by many to be one of the nation’s literacy experts, and was once at U Albany, now at U of Tennessee.  At Southern Westchester BOCES, we used him as a literacy consultant to present while I was a Director of Professional Development.   Here’s the link to the Winter 2013 issue:

http://www.saanys.org/uploads/content/VANGUARDwinter2013.pdf

I’m struck by many of Allington’s points about literacy, about common misunderstandings relating to literacy and the Common Core, and about various forms of supplemental instruction.  Some examples:
-Computerized reading support program research shows little value to students.  Teacher led supportive instruction is what works.  “There isn’t much good to write about computerized instruction.” (p. 26).
-Whole Language also works, when done well, better than phonics only, but later in the article Allington points out that phonics instruction is vital for the 10-20% of students who don’t pick up on phonic awareness without explicit instruction.  These kids need specific help.
-The Common Core is frequently misunderstood — it does not mandate consistently harder texts.  Read his comments carefully to see how he interprets the Core Standards, and how he points out that teachers should be exposing kids to multiple levels of reading.  “There is no evidence that giving kids harder texts will have any positive effects on reading achievement.  Even the CCSS doesn’t suggest this occur….. States did not approve the Advice to Publishers so it is a puzzle to me why so many teachers seems to think harder texts are recommended by anyone.” (p. 24)

Given the reputation that Allington has in literacy instruction, these comments on the Common Core should be somewhat reassuring to teachers concerned that the Core stretches reading levels too high for many students, or that the Core mandates instructional texts at levels higher than appropriate for grade levels.  Allington also suggests some strategies and responsibilities for teacher and districts serving low income students that can support the vocabulary deficits that are found when children first enter schools.

This is an article worth reading and widely sharing.  I hope you find it as encouraging as I did.  Check out the references at the end as well if you can get them — several are not online for free.

Critical Thinking ReDo…

Posted on by
Reply

In an earlier blog On Critical Thinking, I lamented the dearth of analytical skills among the public, and I suggested that as K-12 educators, we have not produced generations of sophisticated adults who can find the truth in the media swamp surrounding them.  On Sunday, March 3, Dominique Brossard and Dietram A. Scheufele, professors of science communication at U Wisconsin, Madison, wrote “This Story Stinks” in the New York Times.  It’s a disturbing account of problems with the Internet as an information resource, and a demonstration of why critical thinking is so important.

The authors asked 1183 folks to read a news post on a fictitious blog.  It discussed a new technology product–nanosilver–with potential benefits like antibacterial properties and risks like water contamination.  Then the readers were exposed to comments written supposedly by other readers.  Half the comments were “civil reader comments and the other half [were] rude ones.” The actual contents of the comments was designed to be equally supportive or opposed to the technology, but the rude comments “contained epithets or curse words.”  To the researchers, “the results were both surprising and disturbing.  Uncivil comments not only polarized readers, but they often changed a participant’s interpretation of the news story itself.”

Based on a preliminary survey of attitudes toward technology, there was no change in pro- or anti-tech views after reading the civil comments.  Among those who read the rude comments, the readers became more polarized in their views of the technology, and made “study participants think the downside of the reported technology was greater than they’d previously thought.”

So what are we seeing here?  This feels like an explanation for the loss of civility so evident in media everywhere–cable TV news and commentary, the extreme polarization of public political discourse–such media trends parallel the findings of these researchers.  As I click through various web links searching for information, I often linger on the comments that follow articles.  I’m frequently taken aback by the nastiness of many who leave comments.  I find political garbage tossed into comment boxes where it doesn’t belong or blather that doesn’t contribute anything factually related to the news post, and I’m often offended that the comment wasn’t moderated and eliminated from the site.  It’s like junk email and it’s irritating.  Apparently, it also unconsciously alters my ability to make the best and most objective use of the information contained in the article–I might just think those who don’t agree with my opinion of the article are simply ignorant idiots, as the rude commentators have so cleverly stated in their posts.

Combine this finding with what we’ve been told about search engines.  Search engines carefully assemble a profile of our individual interests so well that they begin to match our search results to the assembled profile about us.  We begin to get results that are prioritized to match what we’ve been looking at already.  If we don’t go to the last pages of the search results, we will only see hits that mimic what we’ve been reading most frequently.  So, when the engines see we’ve been following Fox news or reading biographies of Paul Ryan, we get conservative results first.  If we follow Jon Stewart, we’ll get more liberal results.  If we read about Michelle Rhee and the Kipp Schools, we’ll get more results favorable to charters and fewer results that might be critical.  Unless we work very hard to defeat the engines, we don’t get the chance to make up our own minds–we are fed confirmations of our prior results.

If the loud noises and rude protests in web chatter interfere with our ability to make proper judgments about news sources, it’s further evidence of critical thinking failures.  The squeaky wheel gets the grease, as all teachers know.  For example, most people think  we wind power is more expensive than coal and natural gas, since those energy industries have been working very hard to convince us that investing in wind power is a waste of money.  But, the truth is that wind power, once you take away all the currently existing government subsidies to the energy industry so that you compare costs on a level playing field, is cheaper per kilowatt hour than coal or oil or natural gas production.  So why don’t we know that with some confidence?  Perhaps it’s because of the background noise and rude chatter that permeates any alternative web discussion board that allows open posts, in addition to media outlets who fail totally at fact checking.  Without dispassionate analysis, Truth is damned–nonsense and misinformation and shouting down the truth sells, and critical thinking is mired in the slough of despond.

 

Value-Added Models – New Research

Posted on by
Reply

If you’re following educational reform, you are aware that some flavor of value-added statistical modeling is being pushed throughout the nation as a means to identify good/bad teachers or good/bad schools.  Pushed now by the US Department of Education and Race to the Top, and promoted by a specialized cadre of assessment developers who stand to rake in massive profits from the testing required to generate data to be used in value-added analyses, this movement has been sweeping across the country.  It has, on the other hand, always raised the suspicions of educational researchers, some economists, and numerous statisticians who have suggested that the models, and the assessments behind them, simply don’t work well as a basis to make high stakes decisions about educational policy, about schools and teachers, or about children.

In January, the Educational Policy Analysis Archives, a peer-reviewed online policy journal, ran several research articles in a special issue covering value-added, called “Value-Added: What America’s Policymakers Need to Know and Understand.”  The title is very misleading, since my reading of this research suggests a better title might be “Value-Added:  What America’s Policymakers Consistently Ignore.”  I’m highlighting the articles here, with links, so readers can pick and choose what interests you.  Each separate article offers a substantial list of references for further exploration.

The opening discussion by the editors is “Value-Added Model (VAM) Research for Educational Policy: Framing the Issue.”  The editors provide background and relevance of six additional papers included in the special issue.

Diana Pullin, of Boston College, a lawyer and legal scholar, offers “Legal Issues in the Use of Student Test Scores and Value-Added Models (VAM) to Determine Educational Quality.”  Given that many states are implementing teacher evaluation models in which VAM measures will be used to dismiss teachers, Pullin analyzes the complexity of predictable court challenges as experts on both sides of the VAM movement offer conflicting testimony about the validity and reliability of tests and the statistics used for employment decisions.

Bruce Baker, Joseph Oluwole and Preston Green of Rutgers, Montclair State, and Penn State respectively,  in “The Legal Consequences of Mandating High Stakes Decisions Based on Low Quality Information: Teacher Evaluation in the Ract-to-the-Top Era,”  review issues around the utility of student growth models and VA models.  Their lengthy appendix outlines the VAM evaluation policies of several states.

Nicole Kersting and Mei-Kuang Chen of U of Arizona, and James Stigler of UCLA, offer a statistically heavy analysis titled “Value-Added Teacher Estimates as Part of Teacher Evaluations: Exploring the Effects of Data and Model Specifications on the Stability of Teacher Value-Added Scores.”  They conclude there are several problems with stability and sample sizes which suggest a need for more work to improve the measures.  If you’re not statistically competent, read the narrative–the supporting details are a second language for many of us.

Elizabeth Graue, Katherine Delaney, and Anne Karch of the U of Wisconsin, Madison provide a qualitative research paper entitled “Ecologies of Education Quality.”  Their analysis suggests that VAMs can’t capture or control for all the variables of effective teaching.

“Sentinels Guarding the Grail: Value-Added Measurement and the Quest for Education Reform,” from Rachel Gabriel of the University of Connecticut and Jessica Nina Lester of Washington State is another qualitative analysis.  This is an interesting review of how proponents of VA suggest the methods are scientific and accurate, while the cautions of educational researchers with significantly opposite points of view are ignored or dismissed.  This analysis of the effective public promotional efforts of VAM and the glib solutions to educational problems that VAM supporters say will come through it’s adoption highlights how public policy has been shaped by politics and the media.

Finally, Moshe Adler of Columbia University, in “Findings vs. Interpretation in “The Long-Term Impacts of Teachers” by Chetty et al.” offers a a refutation of the claims of an often-cited study suggesting that the positive effects of a singe year with a good teacher last a lifetime,  Since the Chetty el al. study is frequently cited as justification of VAM and teacher evaluation changes, this is an important commentary on what Adler says are the false claims of this seminal economic impact study.