Neuroscience Nonsense

Posted on by
Reply

Yes, that’s an in-your-face title, but having written previously about how education isn’t particularly good at using research, but is good at jumping at fads, here are two links that illustrate the misuse of research in education.

Sarah Sparks writes a blog for Ed Week which I don’t regularly read, but after running into this entry I think I’m going to subscribe.  On August 11, she wrote “Making a Mountain of an Educational Neuroscience Molehill,” in which she links to a Nature journal article that takes a critical look at the hype surrounding several educational applications of neuroscience clamining to improve student performance.  The article, “Power failure: why small sample size undermines the the reliability of neuroscience,” offers a somewhat statistically dense explanation of why many of the reported positive effects of various neuroscience educational applications are of questionable value: They come from studies with too few students/cases to be reliable.  This means they can’t be replicated with any regularity, and the likelihood that the results are actually true is not very high.  

Over the years, education has been peppered with a wide range of proposals to improve student performance based on weak research followed by someone’s feel-good suggestions about how our brains work.  Sparks notes the left-brain/right-brain research in her article.  This is among the most popular fallacious notions about learning approaches, very popular among educators, the subject of many books, and unsupported by well-designed neurological research.  Another notion that feels good and sounds good to teachers is learning style research, which suggests that we have to identify a preferred learning style for children in order to design instruction that will be effective for them.  This too, is unsupported by carefully designed research, and might have even more books written about how to implement learning styles than the right-brain/left-brain theories.

That learning styles is unsupported is a shocker to me: It just feels like it makes sense to me.  My doctoral dissertation research was heavily invested in Myers-Briggs type, and my early research looked into several variations of learning styles and communication styles and how they might influence administrative behaviors.  It seems to me that I can identify students from my 20+ years in the classroom that responded differently to changes in my teaching style.  That children can learn in many ways is clear.  That they respond differently to different teachers is also clear.  That this has to do with learning styles is not, in hard-core research terms, demonstrable.

Another ‘movement’ that was going to help our low achievers blossom was the effort to reinforce positive self-esteem among all students, including the low achievers.  Praise, praise, praise.  Always find something to praise even when results were dreadful.  Praise effort over results.  Most good teachers, fortunately, had a hard time doing that, because they knew that low-performing kids understood clearly that they didn’t get it and the praise was artificial.  Fortunately, enough good research evolved over time that I think this sugar-coating of low performance is no longer in vogue.  Kids benefit from successful learning experiences for which they deserve praise, not praise for performance they know is mediocre.

What does this mean in practice?  Many things, I think.  First, while we would like to see education as a science that can be nailed down with research-based explanations for everything so we can teach everyone how to do it, unfortunately it’s not all that easy.  I believe in research to attempt to understand want works, but I also believe that we have to pay attention to research that debunks the ‘innovations’ we emotionally adore.

Many educators want to say that teaching is an art, not a science.  I’m not one of them–there is a lot of science to teaching well.  We know a lot about what works.  Most teachers should be able to use that science to improve their teaching.  That some teachers can take the science of what works and do considerably better than average is where the artistry comes in.  I think it’s an interpersonal talent–an ability for a teacher to connect to students more completely or more consistently than other teachers.  Every teacher can connect with a few students every year: Some teachers connect with more than others.  A few teachers are magicians with kids.  We haven’t figured out the magic yet.  If we ever do, it will transform education.

 

Bruce Baker on Value Added and Student Growth Percentiles

Posted on by
Reply

I’ve lauded Bruce Baker’s blog once already, but day by day I’m becoming an even more devoted follower of his wit and his brilliant exposés of fallacious research, bogus claims, and shallow thinking.  I’ve learned/confirmed so much by subscribing to School Finance 101, particularly about value added models and student growth percentile models, as Dr. Baker reexamines the research on these topics and highlights the shortcomings and the unintended consequences of using these measures in states all over the country.

I’m very interested in his work on student growth percentiles, first created by Damien Betebrenner at the National Center for the Improvement of Educational Assessment, www.nciea.org. First used in Colorado, and now spreading to Massachusetts, New Jersey and New York, the proponents of SGPs suggest they fix the problems inherent in value added measures.  Baker emphatically says not so, and uses the writings of the fathers of SGPs to highlight how SGPs are being misused.

To get up to speed on VAMs and SGPs and their problems as Baker sees them, go through the blog references found in his Value Added Teacher Evaluation category here, and as you read the articles, click on through to the links within each one.  You’ll get quite a wonderful look at the limitations of these statistical models as they relate to teacher evaluation.  Baker also takes on state education officials in several states, including Colorado, New Jersey and New York, with his irrevent commentary.

Of particular interest for New Yorkers is his analysis of the preliminary technical reports on the results of NY’s first SGP assessment results.  This analysis can be seen in the entry entitled “AIR Pollution in NY State…”  He offers graphs showing how factors that cannot be attributed to teachers have an effect on the patterns of SGP scores–students in low income schools generally underperform students in higher income schools, as one already knows, but the SGPs of these students also lag behind higher income schools.  This demonstrates that SGPs do not, in fact, account for effects of peer groups, of school effects, or of poverty, as some suggest.

Read Baker and weep about how assessments are regularly being misused by people who should know better,

Why the Opposition to Testing and the Common Core?

Posted on by
1

I spent the last 20 years of my career dealing with data as a means to improve instruction, and I coached lots of people about how to make sense of data, standardized and local, as a useful component of daily instruction.  As teachers or administrators, we can’t know what our students have mastered without good tests, whether they are formative tests that are not graded, unit tests, interim benchmark tests, or standardized accountability tests.  In my own state of New York, there is a rising movement among some parents, some of whom are teachers, to opt-out of state NCLB tests.  Opposition to ‘national’ testing and the Common Core is growing in other states, supported by various groups who are concerned about increasing federal influence on local educational policies, costs, the misuse of tests for purposes other that what they were designed to do.

The two national testing consortia are talking about designing the tests to improve their utility as vehicles for instructional improvement, but whether these goals will be a part of the basic package of new tests or available at additional cost to local districts remains a bit fuzzy.  States are beginning to question the cost benefit analytics of the programs–both consortia are designing online testing while many school districts, including the low income districts that have always been the targets of the accountability movement, are going to struggle with paying for the Internet infrastructure and hardware required to support online testing.

So in broad terms, the growing opposition to the Common Core and the national testing programs represent multiple issues of concern to a growing variety of stakeholders, and they overlap significantly, making this a very complex situation.  Let’s review some of them here.

1.  Use of data for instructional improvement.  In New York, where at one time all state sponsored tests were released publicly after testing was complete, we could use the results for instructional improvement planning.  We had specific information about how the questions aligned to standards, we could see the question, and we had a p-value which told us how difficult the item was state-wide.  This was very useful for teachers, but that’s gone now that NCLB tests are secure.  Without seeing the questions, the rest of the information (how I did compared to anyone/everyone else) provides very poor instructional information.  I can still see whether I did well or poorly on a particular performance indicator, but I don’t have decent details about exactly what my kids have missed and I have to guess about why.  If I am in a district that has purchased commercial assessments that give me access to the questions, I have this capability during the school year, but many publishers tests are also secure and I have to rely on a generic description of academic trends that isn’t particularly useful.  I get far more information on what a student’s weaknesses are when I can review the test question and analyze the wrong answer responses as I want to plan interventions.

2.  Purpose of the tests.  Critics of NCLB testing write about this regularly.  The tests were not designed for either instructional improvement or teacher evaluation.  They have been co-opted by politicians and businessmen as a means to promote agendas other than school improvement.  Psychometricians who are not involved in developing or marketing these tests have written extensively about their concerns when they are used for high stakes teacher evaluation, or for determining student placement.  We don’t do the latter in NY, as far as I know, though there have been some examples of placement consequences for kids–it’s a local issue.  But in other states, testing occasionally has greater impact–promotion or retention, acceptance in accelerated programs or placement in remediation–based on results.  Often the uses of results doesn’t match the psychometric properties of the tests, which is problematic.  And in New York, where the state says it’s own tests count only for 20 of 100 points in a teacher evaluation, the rubric for the 100 points actually creates situations in which the 20 points on the state test can actually override the other 80 points.  This means everything about New York’s intricate evaluation systems can be blown away by a weak performance on the state 20% measure, meaning this 20% can, for some teachers, amount to the only measure of teacher quality that counts.

3.  Lack of transparency.  I hope this issue will go away with time.  It’s rather difficult to get information about the actual behavior of state tests–how did the results break out by student sub group?  By district demographics?  By years of teacher experience?  By class size?  By SES factors?  How did low income kids do in a district where they are 6% of the population compared to a big city, where they are a majority?

Moreover, everyone involved in rolling out new teacher assessments and new academic goals of the Common Core has been overworked and hard pressed to clarify what is coming.  As we heard repeatedly from the highest levels of New York State Education officials early on, these changes are like an airplane which is being constructed during flight.  Sadly, this is about the closest thing to  transparency I can think of early in the process–there wasn’t much more to be said.  They didn’t know how to get from where they started to the intended goal.

The proper answer to such comments from school boards and the public was often something like this:  “Where is the FAA?  Who approved the takeoff of this untested aircraft in the first place?  There’s too much being done here with no piloting, no research, no shakedown flights. But this concern was shoved under the rug.  New York adopted these changes to get $700 million of federal funds over 4 years, which was less than a half a percent of the per pupil spending on public education in the state.

And this is not just a New York issue, it’s national.  We don’t usually do this kind of take-off in other arenas, but it happens too often in education.  Let’s try the next new idea that some expert says is the solution to our problems, while we ignore the fact that it’s an untested and unproven program or methodology.  Did we know the Common Core was a singular organizational initiative from an educational think tank hired by the Council of Chief State School Officers which has, in effect unilaterally influenced the direction of public education?  Would truly empowered state education officials around the nation have adopted something this massive and unproven if they were not required to do so by other political forces, or if substantial federal funding during an economic downturn didn’t bribe them into agreement with Race to the Top conditions?  Would the public have found this a good use of funds if it were clear that there’s no evidence it would work?

4.  New teacher and principal evaluation systems.  A few districts in my region of New York who closely monitor new evaluation regulations are concerned about how honors class teachers did–they have all the high performing kids, and they think their special ed teachers got higher evaluation growth scores in their state testing measures because a point or two of improvement at the lowest level of performance is easier to get than increases among students already at the top.  This is one of the general criticisms of accountability programs that use any version of growth or value-added scoring.  From state to state, have the data been made available to independent researchers for an objective review? Do we have multivariate analyses available to look deeply at the results?  If not, why not, or when will this happen?  And if we do, so what?  Most states are about to change their tests again, from what they have been using to the new PARCC or Smarter Balanced consortia tests under development.  So the nature of available data is changing from last year, before the Common Core curriculum implementation, to Common Core based testing, and then in two years, to tests from one of the two consortia.  This fundamentally means three versions of accountability tests in as few as 4 years in many states, making comparisons of student achievement a statistical challenge for accountability purposes, to say the least.

5.  Narrowing of the curriculum.  How many fewer art, music, drama, etc. classes are gone, and replaced by supports for ELA and math?  Why does the edu/politico establishment ignore the evidence about the utility of arts and music education and their connection to math and science success?   How many kids have no more recess because they are scheduled for support time?  How many schools have dropped career education options in high schools?  How have the fiscal pressures of the past few years forced districts to make narrowing curricular decisions because of the fear of poor test results?  Is the focus on ELA and math appropriate for all students?

6.  What is college and career ready, anyway?  In the Atlantic Monthly of October 2012, Dana Goldstein wrote an important feature called “The Schoolmaster” about David Coleman, credited by many as the leading creator of the Common Core State Standards.  A nonprofit Coleman founded, Student Achievement Partners, provided the intellectual basis to the Council of Chief State School Officers. If Goldstein’s work is as accurate as it feels, he is far more important than our Secretary of Eduction.  He clearly has had more influence on the direction of US education than any other single individual, likely in our history.  And he’s now the head of the College Board, where his concepts of college and career readiness could transform the nature of SAT tests in the near future.

I happen to agree with the concept of Common Core State Standards, and I think their emphasis on critical thinking skills are long overdue, but defining success with a narrowly defined concept of college and career reading misses some important options for a substantial population of students.  Many educators think this focus rather significantly neglects the notion of career ready–we have been eliminating career and tech education all over, or transforming it into expensive tech honors programs at regional BOCES or Intermediate School or Educational Service Agency  locations as career education begins to emphasize forensics and high tech.  So where will we get our plumbers and carpenters and cabinet makers and auto mechanics and house painters and landscape gardeners?

7.  Local control of education.  Here’s the philosophical issue of the day: Should states/local school districts have virtually given up their role in determining the direction of their children’s education?  Constitutionally, this is a state role (which all but one state, Hawaii, turn over largely to local school boards) and not a federal one.  Though the Common Core is an initiative of states working together, it was not an initiative that involved local professional educators from the beginning.  It was an initiative farmed out to Coleman’s nonprofit business, pushed by the business/political wing of K-12 reformers, gained support from those concerned about the generic failures in urban centers across the country, and then offered as fundamentally a take it or leave federal money on the table proposition to the educational establishment throughout the nation.  The claim that all the states were directly involved in these plans is technically correct, but practically speaking the results were top down impositions, not bottom up reforms.

8.  Lack of funds for professional development.  In New York, teacher and principal evaluation rules mandate districts to provide professional development for teachers in order to be able to terminate them.  Given that schools have to upgrade/update technology infrastructure and hardware to prepare for online testing, and given a new state property tax cap, and given that increasing numbers of districts around New York are facing real economic stress and even bankruptcy, there won’t be any money to provide PD for low performing teachers.  Even a mediocre lawyer can prevent the termination of a teacher because the district’s responsibility for support to a weak teacher will be missing.  So who thought this was a good idea?

Many states claim their test results will be used to guide professional development programs to improve the work of teachers.  However, it’s typical to observe the decline in funding for educational professional development nationwide.  Schools are spending the money on assessments, not on professional development, and the tests themselves, as noted earlier, are both secured and not designed for teacher use to promote instructional objectives.  So two forces disconnect assessments from teacher improvement–funding and inappropriate test design.

Nationally, the movement to tie teacher evaluations to test scores was fueled significantly by Federal Race to the Top eligibility requirements.  This is another ‘reform’ initiative fueled without a quality research base of support.  Proponents of testing, and those wanting to pry public money out of the hands of school boards so that it’s available to alternate commercial programs have jumped on the novel idea that one can appropriately predict a student’s future success based on the scores of a teacher’s students.  The model is appealing, as it suggests an easy metric on which to judge the performance of schools and individual teachers.  But having followed the arguments closely for 10 years, it isn’t working particularly well anywhere.

Today, many groups are finally responding to some or all of these concerns by pushing back.  Some of the parent arguments arise out of fear for the well-being of their children, some out of frustration at loss of local control, some out of growing awareness of rising assessment costs and little demonstrable efficacy in using tests for instructional improvement.  Some opponents are teachers who are fearful of adverse consequences to themselves.  Academics, politicians and the occasional state education official are more openly questioning the speed of implementation, the lack of piloting, the unintended consequences, the public relations disasters in states where Common Core based testing produces a drop in test scores (teachers are not yet trained in the new curriculum objectives), and the mismatch between test design and the use of test scores.

Does the push-back accomplish much?  To date, not really.  Does it make a statement that might force politicians to take a second look at the unintended consequences of parental opposition?  The consequences of testing?  A real cost-benefit analysis of CCSS and national testing?  I personally hope it does, but I don’t think it will amount to much unless it continues to grow.  Get up to speed, and get involved!

We Understand What Works, Why Don’t We Pay Attention?

Posted on by
Reply

Years ago when I was actively writing educational grants for a living, there was a federal program that supported replication of previously funded grants.  It was possible to funded to take the results/methods/findings of successful prior grants and try to replicate those programs in your own setting.  For reasons unknown to me, that program faded out of existence about the same time that federal funds for education grants declined overall, and eligibility rules were tightened to direct more of the limited funds to needy districts.

This, I think, was a great loss.  A few projects that proposed to develop new approaches to problems that had been successfully addressed in other settings continued to be funded.  Replications projects disappeared.  This sad development was not surprising, because we live in a society in which what works is regularly set aside for the next new idea–we have to replace everything with something new.  It’s just not acceptable in American society to keep doing the same thing year after year–the new is assumed to be better.

We see this in the drug industry again and again–they keep looking for new drugs to replace effective ones which are out of patent, and hence less profitable.  So they promote the new drug even when it’s no more, and often less, effective than the old one.  We fertilize fields to keep growing the same crops year after year even when crop rotation continues to be as effective, and in many agricultural settings, more effective and more profitable.  But Monsanto tells everyone fertilizer is the only way to go.  The media and corporate pressure for change, change, change overwhelms the research again and again.

How does this play out in education?  In simple terms, the education industry continuously ignores the research about what works in schools and goes for some new idea, which is often no more than a rehash of an old one in different clothes, and often a bastardization of an effective program that is actually less effective.  We have researchers who must do new research and publish new data for their career advancement, which helps to generate this churn.  We have publishers who are quick to grab new ideas as the next best thing regardless of any evidence of efficacy.  This is ‘corporate think’–new is always better, period.

In a recent Education Week special edition entitled Leaders to Learn From, February 6, 2013, several educational leaders were identified as exemplary folks who made a difference in their schools.  Read about all of these standouts here.  Leadership examples cover 15 topics: Dropout reduction, English-Learners, school turnarounds, transportation, rural enrollment, special education, district-union partnerships, parent engagement, school climate, college readiness, digital learning, social networking, discipline, STEM education and smart growth.

What strikes me about virtually all of these leaders is that their work is not dependent on formal assessments to drive teacher and principal performance.  It is dependent on creative leaders who have paid attention to the research literature about what works, looked at and adopted programs that work elsewhere.  They have adapted those ideas to their local districts, often thinking out of the box, but always taking good ideas that have been known for years and supported the implementation of those ideas in the local environment.

The one exception I see in the special report is Clark Hults, Superintendent of a New York Adirondack Mountains ultra-rural one room schoolhouse district with under 100 students K-12.  He created an outreach program to recruit international students who live with local families, growing his district, generating income that funds programs, and creating a richly diverse student body from an otherwise rural monoculture.  This was truly thinking out of the box to find a unique and creative solution to maintaining programs in a district where consolidation with neighbors would mean hours of school bus time beyond the hours already required– rural here is really rural.

Aside from this example, these leaders are experts at taking proven ideas and implementing them to improve their schools.  This is, I think, a significant accomplishment and overall, an example of where public educators have missed the boat.  We should be paying attention to what works and finding a way to replicate the successes of others.  When I started my Masters program in 1968, my advisor always noted that educational research is about 20 years behind classroom practice.  In some domains, that remains true–in others, the research is almost totally disregarded by the K-12 field.

Yes, that’s an overstatement.  Many K-12 educators know what works, but a combination of lack of funding and political decisions that impose constraints and requirements on schools create mountainous obstacles working against using the research to help students.  As educational leaders, we should try to emulate those among us who buck the system and pay attention to what works.  And we need to promote and highlight the exemplary educators among us that manage to do so, like those highlighted by Education Week last month.  It’s the only way we can turn the tide back to making use of sound educational research instead of the corporate metrics that drive K-12 today.