Category Archives: Background Issues

Non-cognitive skills, Part II

Posted on by
Reply

More than just test scores

Continuing the theme of my last post, I want to reference Henry M. Levin’s paper “More than just test scores,” which can be accessed at http://roundtheinkwell.files.wordpress.com/2012/09/more-than-just-test-scores-sept2012-2.pdf.  I’ve followed Dr. Levin’s work for several years and I’ve enjoyed his contributions to the knowledge base on assessment issues.  In this brief, published online on August 30, 2012, Levin provides an outstanding and thorough discussion, supported by extensive research citations, on the relationship between test scores and earnings.  He notes “…the measured relationships between test scores and earnings or productivity are modest and explain a relatively small share of the larger link between educational attainment and economic outcomes.”

Non-cognitive skills mean career readiness

Much of his paper discusses “the effects that education has on the development of interpersonal and intrapersonal skills and capabilities that affect the quality and productivity of the labour force.”  It was Levin’s work here that prompted me to write this post and my last post on non-cognitive skills, starting with the SCANS work from 13 years ago, which offers substantial background to the reader for Levin’s research analysis.  Just reading Levin’s abstract should be enough to keep you reading to the end of the paper, and then checking out some of the references.

Nowhere does Levin suggest that cognitive skills, as measure by test scores, are not important contributors to economic success later in life.  But throughout the research cited one reads repeatedly that academic achievement is not the only measure of economic success.  One must conclude that focusing 100% on improving test scores as the be-all and end-all for education is a narrow and likely misguided agenda.  Coupled with the SCANS report, it’s clear that a focus on test scores and the corresponding narrowing of curriculum to cognitive issues only is not the best means to improve America’s economic competitiveness or the earnings potential of American youth.

Noting that the discussion of world-class educational standards, and comparisons of nations on various international tests leaves out “other attributes of school or student performances,…” Levin argues that “…basing world-class standards on measures of student achievement is a very limited approach to evaluating any educational system.  To meet the economic, political, social, and personal demand for competency, much more is required of students and adults than just cognitive proficiencies as measured by test scores.”  And he echoes the SCANS work when he suggests that interpersonal and intrapersonal skills are dimensions that can’t be ignored.

Test scores don’t explain earnings

By reviewing numerous studies on the connection between academic achievement and later earnings. Levin adds depth to the common understanding that more schooling equates to higher incomes.  This is correct, but the assumption that the higher earnings are associated with cognitive skills is overstated.  He identifies numerous studies that include test scores in their analysis of schooling and earnings, and they find that the “…impact of schooling on earnings retains more than 80% of its estimated effect on earnings, even after accounting for student achievement.”  (See Bowles, Gintis, and Osborne, 2001, “The determinants of earnings: A behavioral approach,” Journal of Economic Literature, 39(4), 137-1176.)  In plain English, test scores account for “a minority fraction” of the relationship between earnings and schooling, and other elements of schooling “must be accounted for.” Taking on the claim that test scores have predictive validity for worker productivity, Levin cites research that shows scores account for only 6% of the variance in productivity, “leaving 94% to be explained by factors other than the test result.”

So, it seems to me, the research isn’t particularly supportive of pushing exclusively for high academic achievement to return America to world-class productivity.  And besides, the US worker is already among the most productive anywhere, even when our test performances appear mid-range.

Graduation rates do equate with economic performance

The relationship between finishing high school with a diploma and increased lifetime earnings is well-established.  Levin offers some depth to this discussion by pointing to research on five educational interventions to prevent dropouts, which offer net benefits to taxpayers through public heath savings, crime savings, and increased exhale revenues from earnings.  Interestingly, he also notes that GED graduates do not show the same benefits–their test scores are similar to graduates, but their earnings were inferior.

Having shown that cognitive skills and tests scores are not very good predictors of economic performance, but high school completion is, there is something other than academics of value for students who stay in school and complete a diploma other than compiling academic skills.  And these factors, not currently tested, play as important a role, or a greater role, in helping students become ready for the world of work.  I strongly suggest you read the rest of Levin’s article, in which he reviews what those characteristics are, and how schools can improve the way they are supported.  In fact, Levin identifies several programs that teach the social and emotional skills, and cites  research supporting the efficacy of doing so.  He spends some time on the SEL movement–social and emotional learning–and goes beyond identifying the employment benefits by identifying research that shows SEL program intervention “…were comparable to or exceeded the results found in the literature for improving student achievement.”  This is a really significant finding–“…interventions to develop social and emotional skills have particular salience for improving student achievement.”

I’ll end with a few sentences from Levin that should inspire you to read this piece several times, and read some of the back-up research he has so capably assembled.  “The almost singular focus on test score performance in educational assessment at both domestic and international levels does rest on some foundation…. But the evidence does not support the assumption that cognitive skills are all that counts and that they alone can produce healthy and productive adult personalities.”   “Far from being harmless, the focus on test scores and the omission of the non-cognitive impact of schools can create far-reaching damage.”   “Other goals may be as, or more, important in the long run in terms of creating productive, equitable, and socially cohesive societies and economic growth.” (See Gradstein and Justman, 2002, “Education, social cohesion, and economic growth.”  American Economic Review, 92)4_ 1192-1204.

As educators and activists for meaningful educational policy, we have to stop simply looking at easy solutions and start listening to what employers say they want.  Listening to employers was a novel idea when the SCANs report came out in 2000, and it’s likely that those who publish achievement tests and those who push them as easy measures of future productivity will continue to ignore employers in the future.  Real educators should speak up.

Non-cognitive skills can’t be overlooked! Part I

Posted on by
1

Better Education Overseas Doesn’t Explain Outsourcing

Reformers are obsessed with cognitive measures of student achievement.  The reform movement maintains that ELA, math and science scores must be raised in order for American to be a world class competitor on the international scene.  I recall this same theme being promoted in the early ’70’s when New York began the Regents Action Plan to improve test scores.  I listened to state ed department experts justify higher standards in social studies by saying that if our students didn’t know more history, they couldn’t compete with other nations in manufacturing.

That was bunkham, and its essentially the same argument in replay we’re hearing today.  I earned much of the cost of my college education working in a skilled labor setting building machinery that went into automobile production factories.  I rubbed elbows with auto workers.  The skills they needed to manufacture cars had nothing to do with history or social studies, and for that matter on ELA, science or math.  Today, the low wage earners assembling iphones for Foxconn or sewing our clothes in Bangladesh need not be writers or mathematicians–they need to be willing to work in conditions that are, frankly, abysmally unsafe, without benefits, and accept that they have no recourse.  In other words, American jobs have been lost to countries with working conditions Americans would consider slave labor if they were found in U.S. factories.  The educational achievement of Americans is not the issue here at all–profits are the issue.

SCANS Skills–What do Employers Want From Workers?

Now that we are testing the life out of schools at all levels, what evidence is there that cognitive skills might translate into better jobs?  Let’s start this exploration with a look at the skills employers want their workers to possess.  Since schools are eliminating courses, programs, or activities that promote noncognitive skills, what role do noncognitive skills have in the lives of successful wage earners as described by employers?

We should take another look  at a report that came out in 2000, when the U.S. Departments of Labor and Education formed the Secretary’s Commission on Achieving Necessary Skills (SCANS) to study the competencies and skills that workers need to succeed in today’s workplace. The results became What Work Requires of Schools: A SCANS Report for America 2000.  The link above provides a summary of those skills, which start with five competencies, each of which is broken down into additional elements that I’ve not included.

  1. Resources: Identifies, organizes, plans, and allocates resources
  2. Interpersonal: Works with others
  3. Information: Acquires and evaluates information
  4. Systems: Understands complex interrelationships
  5. Technology: Works with a variety of technologies

Examining the detailed breakdowns of these competencies will enlighten the reader–they are virtually all skills that are not taught in schools when the emphasis is on passing state or federally mandated tests.  The information competency comes closest to something we emphasize in K-12 programs today, but the SCANS versions are side-effects of the K-12 content rather than primary goals.

In a second level of SCANS skills, also found in the summary link above, a three-part foundation for SCANS skills is suggested.  I’ll list the three foundations here–italicized elements should be a part of Common Core competencies, and in red, I’ll identify those skills that are not built into the implementation of Common Core curricula that can be tested.

SCANS Foundational Skills

Basic Skills:

Reads, writes, performs arithmetic and mathematical operations, listens, and speaks

Thinking Skills:

Thinks creatively, makes decisions, solves problems, visualizes, knows how to learn, and reasons.  One might argue that Common Core curricula is moving toward these kinds of skills, but when you examine the detailed descriptions of each of these skills, you will see that they are, except for solving problems and visualizing, not particularly testable, and hence are unlikely to be a part of the curricular changes that the CCSS will inspire.

Personal Qualities:

Responsibility, self-esteem, sociability, self-management, integrity, and honesty.  There’s nothing here that is a direct focus of the Common Core, nor are these qualities testable with measures of academic achievement.

Looking at the foundational elements of SCANS skills, you’ll see that employers have identified 17 foundational skills, 10 of which are not academic measures and are not likely to be a part of Common Core curricula or Common Core tests because they simply are not measurable in any easy way.  They are skills that might be incidentally and indirectly learned in schools, but are infrequently or never explicitly taught.  When the SCANS report first came out, there was considerable interest among those who were involved with Career and Technical Education–folks had to show how SCANS skills were incorporated into their CTE programs.  Sadly, with cognitive skills taking the main role in testing, SCANS skills and CTE programs are on the back burner once again.

Social and Emotional Learning

However, there are lots of programs available that explicitly attempt to teach these skills and that have a reasonably strong evidence base supporting their use in schools.  The social emotional learning movement (SEL) is among the best researched.  Find out more about SEL from edutopia.org.  At the bottom of this edutopia page you’ll also find organizations that promote SEL. 

If we can wrest education reform away from the profiteers and politicians, perhaps we can balance our K-12 goals to reinstate noncognitive education once again.  When is the reform movement going to pay attention?

 

 

 

Teachers: The Courts Might Not Care About Bad Evaluation Systems

Posted on by
Reply

Courts and Teacher Evaluation Systems

Writing in Education Week, September 18, 2013, James Popham and Marguerita DeSander suggest that “thousands of American teachers will lose their jobs in the next few years because of the recently designed, more demand evaluation systems now found in most of our states.”  They point out these systems have been triggered by recent federal policies, and then note that most teachers think the dismissals can be reversed in court.  The main point of their essay is that the courts, both state and federal, have “historically refused to substitute their judgement for that of a school board in cases where a teacher (whether tenured or probationary) has been terminated for substandard job performance.  This has been true if the termination is based on even a scintilla of evidence.”

Popham is a well-know figure among education professors who are experts on evaluation and assessment, with several books, most recently (2013) Evaluating America’s Teachers: Mission Possible? from Corwin Press.  DeSander is an ed admin professor and formerly was an attorney specializing in employment law.  These are two folks who should be seen as authoritative figures on this topic.

New York Teachers More Protected?

When I first saw the outlines of New York’s response to Race To The Top teacher evaluation changes, I thought that it might be a good time for college grads to go into the practice of education law, particularly if they had any statistical talent and were willing to delve into the psychometrics of testing and its use in teacher evaluation.  As New York’s teacher evaluation system, including the use of test scores in teacher evaluation rolled out, I continued to believe the system would collapse under the weight of court challenges to it’s fairness and it’s validity.  I still believe that, but I’m not quite so confident with my opinion after reading this piece.  If I look at New York’s system, which has a unique additional element–teachers identified as low performing have to be given opportunities for professional development by their school districts–perhaps that district responsibility is going to prevent willy-nilly teacher dismissals.  With the greater than anticipated increased costs of testing, and the Governor’s imposition of a 2% tax cap on raises in school budgets, it’s already estimated that 40% of districts across the state will be in bankruptcy in the next few years.  So districts won’t have the money to provide professional development to their low performing staff, meaning they might not be able to dismiss them regardless of evaluations.  That’s not the case in other states, where these warnings may be far more telling.

Evaluation Systems Show No Evidence of Efficacy

Popham and DeSander recognize that across the nation, the new evaluation systems have serious flaws–relying “too heavily of traditional achievement tests…”  They note that the tests “are unaccompanied by any evidence that they are able to distinguish between well-taught and badly taught students.”  They also point out that courts, based on past patterns of refusing to rule on the merits or validity of evaluation systems, “will not rule on the appropriateness of a teacher-evaluation system, or the evidence-collection procedures incorporated in that system, if the procedures are applied in a fair and consistent manner to all teachers affected. Thus, even an inadequate evaluation system will avoid the rigor of court scrutiny when it is applied equally to all teachers.” (Emphasis mine.)
The authors also take issue with the validity of classroom observations when they suggest that rating teacher classroom performance on the bases of “a handful of 30-minute classroom visits..” and even when looking at 40 or 50 dimensions of classroom performance, “the resultant observation data are often of little value.”

Courts May Not Overturn Bad Evaluation Systems

So where does that leave teachers all over the nation?  I agree with the authors that the evaluation systems lack validity as they are being designed–the reliance on testing is not warranted psychometrically at all, nor are the tests designed for the purposes being implemented.  And if courts are likely to disregard expert opinion because they won’t decide on system appropriateness, there is no venue for expert testimony about why these systems should be disregarded.  If the legitimacy of procedures is not about to be adjudicated, there’s no place for teachers to turn.  One need only review the multiple interpretations of the evaluation designs across 696 districts currently approved by the New York State Education department to know that there is no consistency in their format across the state.  If courts give deference to school boards, teachers might be in far more trouble than they currently expect.  Finally, a good civics education will remind all of us that courts are not always about right and wrong–they interpret the law. Where courts defer to school boards and states who have implemented bad evaluation systems, teachers beware.

Algebra II Revisited

Posted on by
Reply

Algebra II Revisited

There’s a new (May, 2013) and, I think, very important report out from the National Center on Education and the Economy entitled “What does it really mean to be college and work ready?”  The organization and been around since 1988, and reports its work as “…researching the world’s best performing education systems to unlock their secrets.”  I strongly recommend you read it.  Here’s a video commentary on the report by their president, Marc Tucker, video commentary from http://www.ncee.org/college-and-work-ready/  Insert

I wrote about Algebra II and the Common Core in this previous blog, and NCEE’s research offers another commentary on the value of Algebra II as an indicator of being college and career ready.  I’ll offer some quotes from the report to expand my prior argument against Algebra II for all.

National Center for Education and the Economy Math Findings

NCEE opens their report, which actually covers both ELA and Mathematics college and work ready issues, by noting that 45% of students entering higher education for college degrees or for career preparation enroll in community colleges.  Hence, NCEE argues, looking deeply at the requirements for success in community colleagues is a practical and a real world indicator of what it means to be college and work ready.  About half of community college students go on to four year programs, further supporting their rationale for looking at community colleges.  The rest who graduate are taking a training program that leads to a career option.  The study examined “the most popular and diverse programs….” (p. 2) in randomly selected community colleges in seven states:  Accounting, Automotive Technology, Biotech/Electrical Technology, Business, Criminal Justice, Early Childhood Education, Information Technology/Computer Programming, and Nursing, plus the General Track.  For math, they looked at the math actually taught in the “initial credit bearing courses” in the programs (p. 2).  They looked at the textbooks, graded student assignments, tests and exams to “analyze the reading and writing skills that are required…” in both math and English Composition.

The findings were surprising.  Only one program required first year students to know Algebra II.  NCEE notes that Algebra II content is required for students who will take calculus or use calculus in their work, but citing a 2011 study from the Center on Education and the Workforce at Georgetown, NCEE notes that only about five percent of the working population needs to understand calculus.   “Indeed, community college first year programs of study typically assume that students have not mastered Algebra I.  The most advanced mathematics content used in the vast majority of the first-year college programs we analyzed can reasonably be characterized as the mathematics associated with Algebra 1.25, that is some, but not all, of the topics usually associated with Algebra I, plus a few other topics, mostly related to geometry or statistics (p. 3).”  They continue by noting that the math that ‘most enables” student success is middle school math, ‘especially arithmetic, ration, proportion, expressions and simple equations (p. 3).”

They observe that most high school grads don’t actually have good command of middle school fundamentals, and that competency in math that is rarely taught at any level — “schematics, geometric visualization and complex applications of measurement (p.3)” are also competencies that would lead to college success.

Algebra II is not a college and career prerequisite

Their conclusions on pages 4-6 make this clear statement about Algebra II:  “Mastery of Algebra II is widely thought to be a prerequisite for success in college and careers.  Our research shows that that is not so. The most demanding mathematics courses typically required of community college students are those required by the mathematics department,…but the content of the first year mathematics courses offered by the …. mathematics department is typically… Algebra I, some Algebra II and a few topics in geometry…. Based on our data one cannot make the case that high school graduates must be proficient in Algebra Ii to be ready for college and careers.”

They further conclude that rushing students into Algebra I early misses the opportunity to emphasize the very strong level of content mastery that should be obtained at the middle school level before moving on to higher math.  They suggest that understanding basic concepts at higher levels than now expected would permit students ‘to learn whatever mathematics they need for the path they subsequently want to pursue more quickly and easily than they can now( p. 4).”

More high school math options needed

Another noteworthy conclusion (p. 5) is “American high schools should consider abandoning the requirement that all high school students study a program of mathematics leading to calculus and instead offer that mathematics program as one among a number of options available for high school students … with other options available (e.g., statistics, data analysis and applied geometry) that include the mathematics needed by workers in other clusters of occupations.  By doing so high schools will almost certainly expand opportunity to many students who now find success in college closed off by a one-size-fit-all sequence of mathematics topics that actually fits the requirements only for a very narrow range of occupations.”

These are powerful words that fly in the face of Common Core standards.  But if the CCSS are expected to prepare students for college and careers, who determined the specifics of these standards?  When one considers the folks who wrote the math standards, one sees individuals highly vested in core math content who seem to have taken highly competitive college admissions criteria as their basic and their only competency target.  They ignored the career level requirements of most workers in the nation, despite clear research about what career mathematics competency standards should be.

Let’s have a look at job requirements.  I’ve read elsewhere (See Andrew Hacker’s New York Times 2012 opinion piece or this commentary from Get Schooled, by Maureen Downey.  This comment excerpt from penguin mom on the Get Schooled piece reproduces my argument rather well:

I was a math/computer science major in college. I did use what I learned in my Algebra 2, Trig, and Calculus high school classes… in college. Once I got out of college, I did Not use that information again even when working in the computer field. You do not need to know logs or rational equations or trig to write a cash register program or pretty much any program besides an engineering one. I didn’t pick Algebra 2 back up until I entered the teaching arena and started teaching the concepts again. Besides the general logic skills learned from solving the problems, I really don’t know of a lot in Algebra 2 that will translate into real world usage for the vast majority of the students. Graphed a line or parabola recently anyone? Used the quadratic formula? Solved a Trig equation? I think it would be Much better to require strong Economic, Personal Finance, Business Accounting and (usable) Statistics courses for every student. That information would be used in just about any field. Figuring interest or profit. Understanding (at least somewhat) what poll numbers actually mean. Keeping a budget and balancing a checkbook. Those are all useful skills.

Get this NCEE report, read the math section, and if you are a generalist, read the English Literature findings as well.  These are a topic for another time.