Because Girls Can't Do Math

Can girls do math as well as boys?  Deep down inside, a lot of girls, parents, and teachers still don't think so.  A recent study by Catherine Riegle-Crumb and Melissa Humphries at the University of Texas at Austin found that high school teachers rated the ability of white female math students as lower than that of white male students with comparable test scores and grades.  In the study of 15,000 students, teachers were more likely to rate courses as being "too hard" for the girls enrolled in them.

Internalized gender stereotypes about who can and should do math develop early - as early as second grade, according to a study from the University of Washington.  In the study, 247 children worked at a computer to sort boys' names, girls' names, and math or reading-associated words.  They were quicker at their task when asked to associate boys' names with math words than when asked to associate those same math words with girls' names, indicating an implicit sense that girls and math just don't belong together.

Where do these stereotypes come from? It turns out that young girls are pretty good at picking up the beliefs around them. One study has shown that when female elementary teachers are anxious about math, their female students quickly internalize the belief that math is not for girls. As girls incorporate that belief, their achievement starts to lag.

But even when girls do well in elementary school math, it's true they have trouble keeping up in high school, isn't it?

Well, no, it's not. In 2008, Janet Hyde of the University of Wisconsin led a team studying the SAT and grade 4-10 math test results of 7 million US students.  Whether they looked at average scores, scores of the most gifted students, or ability to solve complex problems, they found no significant differences between boys' and girls' achievement.  The one place where boys outscore girls is on the math SAT, but this can be explained by sampling bias.  Only kids who mean to go to college take the SAT, and these days that means more girls than boys take the test.  The larger sample of girls dips more deeply into the achievement spectrum, bringing down average scores.  At the same time, nearly half of all bachelor's degrees in mathematics go to women.

The persistent, often unconscious belief that girls just don't measure up in math inspired Todd Bearson, middle school math teacher and composer, to write a satirical song called "Because Girls Can't Do Math."  It's the first song in the Lost in Lexicon musical now under construction.  Here's a sampling of the lyrics:

The female brain is a delicate flowerWhen exposed to math, it wilts and cowersThe way that they think means they don’t have the powerTo solve the kind of higher mindProblems we endure Because Girls…Can’t do math
 Explaining numbers to girls is a waste like no otherWhy not teach a camel to sing?It can’t be done – why do such a thing? Women have a harder time with logicAll their emotions just get in their wayAnd they haven’t a shot at an abstract thoughtTo let them start would mock the artAnd lead us all astray.

But don't worry, it gets better when the girls strike back!  If you'd like an mp3 of the song and a copy of the lyrics with brief notes about the women mathematicians mentioned, send a note to pnoyce@noycefdn.org with the title line, "Send Song." You'll enjoy it.

New Frontiers in Formative Assessment

Formative assessment is an ongoing process of checking students' understanding and then adjusting instruction to best respond to student needs. This process of checking and meeting students where they are takes more teacher skill and flexibility than just marching through a pre-set curriculum, but it should lead to greater learning.

Today is the official release date for New Frontiers in Formative Assessment from Harvard Education Press. Almost two years ago the Noyce Foundation asked me to help share some of the best assessment work of our staff and grantees. I thought a book bringing together interesting work in the field might do the trick, so I put together a proposal, convinced the Harvard folks, and found a co-editor, Dan Hickey, who is an expert on assessment.

Together we solicited authors for twelve chapters on assessment projects and practices. Half the chapters describe the use of technology to aid assessment, often building it right into a learning program. Six more use simple paper and pencil tools to get at students' thinking.

The chapters are evenly divided among math, literacy, and science. It's our contention that different subjects require different approaches to assessment, and we provide plenty of real examples from the projects our authors have run. One chapter discusses techniques for teaching science vocabulary to young English language learners. Another examines how students can label and share graphs among networked computers in a classroom, and another describes a process for preparing teachers to look collaboratively and with diagnostic acumen at students' work in mathematics.

Lorrie Shepard of the University of Colorado at Boulder wrote an insightful foreword, discussing how testing can harm learning and how the chapters of the book provide an alternative. And Dylan Wiliam, assessment maven, gave us this blurb:

This is an extraordinary book. The chapters cover practical applications of formative assessment in mathematics, science, and language arts, including the roles of technology and teachers professional learning. I found my own thinking about formative assessment constantly being stretched and challenged. Anyone who is involved in education will find something of value in this book

.
Working with the authors of these chapters was a privilege, and I'm proud of the book that resulted. I hope that teachers, curriculum developers, and other educators will find it useful. In coming posts I'll give short summaries of selected chapters.

Useful math

To many people it seems obvious: mathematicians should determine the math curriculum. After all, they're the ones who really understand the subject. Part of the reason for a backlash against "reform math" in the 1990s and early 2000s was the complaint that the reformers were math educators, not "real" mathematicians. They might be undercutting important topics because they didn't truly understand where all this math was heading.

But mathematicians are a rarefied breed. Outside of higher education, the number of jobs for mathematicians in the US is approximately 2900. In ten years the number is expected to be approximately 3600. That's a large percentage growth, but a small absolute number. Even when you throw in jobs opening in higher education, there are not enough to accommodate the number of mathematics PhD's being produced. There just aren't that many professional mathematician slots - and certainly not pure research mathematician slots - for students to grow up and fill.

On the other hand, there are hundreds of thousands of new jobs appearing for computer programmers, modelers, economists, accountants, businessmen, actuaries, engineers, statisticians, and people who use sophisticated mathematics to understand fundamental physics, how the genome is organized, potential damage from earthquakes, the workings of the derivatives market, or the geometry of antiviral drugs. That is, there are plenty of positions for people who graduate with strong mathematical understanding and the ability to apply it well.

When it comes to deciding what math is really important for people to know, perhaps the viewpoint of those who use advanced mathematics to accomplish practical goals is as important of those who love and study math only for its intrinsic beauty. By all means, let the mathematicians weigh in about what our children should learn - but include the applied mathematicians. And include practical mathematics. Here's an article by a friend of mine arguing about what that practical mathematics might look like.

Raising the prestige of the teaching profession: can we do it?

At the end of last week I had the privilege of attending (as an observer) an international summit on the teaching profession in New York City. The summit, hosted by Secretary of Education Arne Duncan, invited delegations of 15 countries that are either high performers or fast improvers on the international PISA exam. PISA tests 15-year-olds on broad measures of literacy in reading, science, and mathematics. US performance on PISA, as on other international exams, is mediocre.

The purpose of the summit was to discuss strategies for building a high quality teaching force. All the countries agreed that the quality of teachers has the biggest in-school impact on student learning. Virtually all of them agreed that one important way to build quality in the teaching force is to build the prestige of the teaching profession.

Building the esteem in which teachers are held creates a virtuous circle. Top-performing countries such as Finland and Singapore draw their teaching candidates from the top third of their high school classes. In Finland, there are ten applicants for every available space in national teaching colleges. Both countries work to make sure that teaching salaries are competitive with salaries of other college graduates. In Singapore, teachers are paid a competitive stipend while in training, along with regular bonuses and performance bonuses while they teach. In both countries, teachers are given a lot of autonomy. They mentor and evaluate one another to maintain high quality within their schools.

So the cycle becomes: stipends while in school and competitive salaries afterward ---> highly selective entry criteria ---> highly qualified, ambitious teachers ---> training and opportunities for advancement ---> teaching becomes a sought-after and respected profession, leading to stipends, high salaries, good working conditions, rigorous peer standards, and top applicants.

Singapore and Finland are small countries with centralized decision making and only one (Singapore) or eight (Finland) teacher training institutions, so common decisions about standards and compensation are possible. In the US we have hundreds of teacher training institutions along with alternate pathways; teacher training is often regarded as a low-cost "profit center" for universities, which motivates them to keep entrance requirements low and enrollments high. We train more elementary teachers and gym teachers than we need while we have systemic shortages of qualified math, science, and foreign language teachers.

The public knows that many US teachers come from the lower half of the class. Because the public lacks faith in these teachers' knowledge and professional skills, we often ask them to adhere to a lock-step curriculum. We spend lots of time trying to figure out how to weed out bad teachers, we imagine that they don't work hard, and we resent their benefits. Who would want to become a teacher?

How can we change this picture, especially in a time of constrained funding? One thing I think experience shows is that raising the bar to entry makes a course of action more attractive to high performers. Being chosen for Teach for America, which accepts only about ten percent of applicants, has become a mark of prestige for high-achieving college graduates. The same applies to the rigorous Boston Teacher Residency, a year-long, stipend-paying master's degree program in teaching that accepts only 13% of applicants.

After the conference, my family took the subway to Brooklyn to see a play. We got talking with a young African-American woman who sat near us. It turned out she was a math major who worked in industry doing statistical analysis for a couple of years before deciding to go into teaching. She was recruited by Math for America, a non-profit that is working to build a corps of excellent math teachers in cities around the country. This young woman is receiving a full scholarship along with a stipend during her training year. Then she'll receive mentoring, professional development, and a supplemental stipend during her first four years of teaching in New York City.

This young woman, who is going to be a middle school math teacher, was bright, bubbly, and enthusiastic. She bonded quickly with my seventh grade son. He would love to have her as a teacher. Coming out of Math for America she should have no trouble being hired, and she has a good chance of becoming a leader in her profession. I already hold her in esteem.

I used to think programs like Math for America were an expensive, piecemeal approach to improving teaching and learning in the US. Now I'm beginning to think that these programs may be the pilot lights showing what we can accomplish if we focus on recruiting, supporting, and rewarding teachers of the highest quality.

[For more on this topic, check here.]

Professional Development Does Work, at least in math

Five years ago I wrote a commentary called "Professional Development: How Do We Know If It Works?" for Education Week. In the article, I argued that the only proper measure of success for teacher professional development is whether their students learn more as a result. I pointed out that in the extensive literature on the "best" approaches to improving teachers' skills, there was vanishing little evidence that interventions lead to increased student learning. Mostly this was because the people planning, paying for, and delivering the teacher training were not gathering the data needed to see if it helped students learn more.

Now, however, evidence is beginning to accumulate, at least in the area of mathematics. For a long time, intuition has told us that teachers who know more mathematics must be teaching it better, and indeed, studies have shown that secondary students whose teachers have advanced degrees in mathematics learn a little more than those whose teachers don't. But now (actually, in June 2009)Rolf Blank and Nina de las Alas of the Council of Chief State School Officers have done a meta-analysis of multiple studies that shed some light on what professional development can do to improve student learning.

Blank and de las Alas combed through all the studies they could find on professional development in math and science over a 20-year period. Eventually they found 16 well-designed US-based studies in math and science PD that included control groups (students whose teachers did not get the training) and good outcome measures. Twelve of the 16 studies focused on math. The researchers also identified features common among many of the interventions.

Their findings? In general, the effect of math professional development was positive but modest, with effect sizes averaging .21 for differences in pre-post test growth. That's enough to take an average student from the 50th to the 58th percentile. Effects were greater on test measures that closely matched the training and lower on more distant tests such as state tests.

These successful PD programs tended to share several features. They were focused on math content and how to teach it. The programs were long, averaging over 90 hours, with most of them spread out over six months or more. Included in the PD was a program of follow-up and reinforcement, including support from mentors, coaches, and colleagues to help teachers take what they had learned into the classroom. Effects for elementary teachers tended to be higher than those for secondary teachers.

The report has lots of meaty detail, and is well worth a read for those who want to design and carry out the most effective and cost-effective professional development. But for those who are not statistics nerds, it's good to know that we are finally accumulating rigorous evidence that the money school districts spend on programs to improve teacher skill really can pay off in increased student learning.

High School Calculus Leads to . . . What?

High school calculus opens the door to careers in engineering, physical science, statistics, and all sorts of other math-heavy careers. That's what we've been told. And to be able to take calculus by grade 12, we need to make sure our kids are taking Algebra I in 8th grade. Yep, that'll do it. Hurry along, and we'll get ourselves a lot more scientists and engineers.

Except it's not working that way. Professor David Bressoud of Macalester College, just past president of the Mathematical Association of America, has been talking about this problem for several years. He points out that even as more students study and pass calculus in high school, college enrollment in higher level calculus and other advanced mathematics classes is staying flat or dropping. The percentage of US bachelor's degrees awarded each year in math-intensive majors has been falling since 1984. High school calculus may actually be discouraging students from pursuing science, mathematics, and engineering careers.

First, some facts. About 20% of high school seniors, or a full third of those going on to college right away, now take their first calculus class in high school. That's actually MORE than the number taking Calculus I each year in college. But only about 7% of bachelor's degrees, or maybe 110,000 BA's a year, are awarded in the math-intensive majors of physical science, mathematics, statistics, or engineering.

Of those taking calculus in high school (around 600,000 students a year), about half take the Advanced Placement calculus (AP) exam. The others either take AP calculus courses but skip the exam (perhaps because they don't expect to pass), or they take a non-AP class with "calculus" in the name.

Twenty-five percent of the 300,000 taking the exam, or about 75,000 students a year, earn high scores and receive college credit for calculus. These students--those who receive a score of 4 or 5 on the Calculus AB exam or a score of 3 or above on the BC exam-- do well. They are likely to succeed in more advanced math courses.

But what about the other 525,000 students a year who took high school calculus? Well, a third of them never enroll in any college math. They made it through that high school course, and now they're done. No more math, ever.

Another third place into pre-calculus or College Algebra. They move back two courses from where they thought they were as they finished high school.

Another 1/6 of these students fare even worse. They're kicked all the way back to remedial math. Their math skills are judged so shaky they end up paying for and taking a math course that doesn't even count for credit.

A lot of these students really thought they were doing okay in math. They worked hard. They did the homework. They made it through. They even took calculus! A friend has told me that the average high school math grade for students who end up placed in remedial math at Cal State University is 3.1, a good solid B.

What's to be done? Professor Bressoud suggests that we still have a lot more to investigate. What are the criteria for placing students in high school calculus? What are they actually studying when they get there? How are college math departments adjusting to students who have some exposure to calculus but aren't ready yet to go on? What happens to the motivation of those who don't receive college credit for it?

I would add some questions of my own. Are other countries, including those with centralized curricula, experiencing similar problems? Are programs like Agile Mind*, which uses technology to support the classroom math teacher, making a difference? And finally, what can we say about the experience of students who choose a non-calculus path, taking a course in Statistics or Mathematical Modeling their senior year of high school? Is it possible that another year consolidating skills in algebra or problem-solving might lead to greater success later on? Might experience exploring newer areas of mathematics like discrete math, probability, game theory, graph theory, or bioinformatics actually do more to prepare and motivate students than the competitive rush to calculus? We need to find out.

*Disclosure: The Noyce Foundation helps fund Agile Mind.

Grand Challenges in Math and Science

What are the grand challenges in mathematics and science that can inspire the next generation? In the sixties it was the challenge of space and racing to the moon. Now our challenges seem at the same time so much closer to home and so much more daunting: climate change, energy needs, an aging world population, and the threat of nuclear terror, to name just a few.

Our business and economic leaders have been touting the need for a better-trained workforce in science, technology, engineering, and mathematics, the so-called STEM subjects. But their message about competitiveness is not getting through in a personal way. The Public Agenda Foundation continues to find that although most Americans agree that there will be plenty of opportunities for jobs involving math and science in the future, more than half of parents say the amount of both their kids are getting in school now is plenty good enough. Meanwhile, the American College Testing service reports that fewer than half of graduating high school seniors are prepared for college math and just over a quarter are prepared for college science. At the same time, 70% of Americans say there's no need to start studying sciene until middle school. But we know that by middle school, kids are already sorting themseves into those who do science and those who don't. Without science in school, how are they going to be inspired to think of themselves as potential scientists?

These questions and dilemmas are some of the reasons the Noyce Foundation has chosen in the last few years to focus on informal or out-of-school science, where we hope kids will find something to excite them. The bigger question of what it will take to motivate and inspire today's students is one I addressed in my keynote address last week at a conference sponsored by the Iowa Math and Science Education Partnership and attended by over a hundred higher education faculty who prepare math and science teachers.

You can find a link to my Iowa presentation here.

Math Anxiety: Can a Math-Anxious Parent Raise Math-Confident Kids?

People didn’t talk about math anxiety when I was in elementary schools. There were just kids who were smart at math and kids who were not so smart. Smart kids got the answer right away; dumb kids didn’t try. We all knew boys were generally better at math than girls. After all, they had the math gene.

The further you went in math, the more likely you were to slip into the not-so-smart category, the ones who didn’t understand right away. The teacher might sigh and roll his eyes; the other kids might glare or snicker or make comments under their breath.

No wonder we grew up into a math-anxious nation. Math teaching guru Marilyn Burns estimates that up to two-thirds (aak, a fraction!) of American adults suffer from math anxiety. For some of us, the anxiety is severe. Sit us down for a math test and we start to sweat. Our pulse and blood pressure rise. We can’t focus or remember; our minds go blank. I’ve never seen this before! What if someone finds out how stupid I am? Such levels of anxiety preclude certain careers and can even make us avoid thinking about budgets, markets, or college savings plans.

Even the milder forms of math anxiety are something we can easily pass on to our children, leaving them as handicapped as we are. So what’s a parent to do?

Those who treat math anxiety in college students suggest that we should first confront a number of harmful myths about math. These include:

Math ability is something you’re born with. You just have it or you don’t.

Sure, there are math geniuses, probably a few every century. But most math ability, like most violin-playing ability, comes from practice. (Math geniuses practice math most of all.) The secret to learning math is to find ways of learning that are effective, and then to keep at it. After all, we work on mastering the English language just about all our waking hours. We should expect mastering the language of mathematics to take some time, too.

Boys are just better at math. But that’s all right, because girls don’t need math as much.

Boys do a couple of points better than girls on national tests like the NAEP, and a full 35 points better on the SAT. But girls do better than boys in high school and college math classes, and earn 47% of all math BA’s. Women are taking their place as doctors, biologists, chemists, engineers—and they need math.

Math is about getting the right answers fast.

Timed tests and some school contests perpetuate this notion. But English mathematician Andrew Wiles spent nine years on one problem. When he succeeded and presented his proof of Fermat’s Last Theorem, he became one of the most celebrated mathematicians of his generation.

There’s no room for creativity in math.

Real math is not primarily about crunching numbers. We have computers and calculators for that. Math is about ideas, finding underlying patterns, thinking of new ways of approaching a problem. More and more school math is beginning to mirror this reality, rewarding creativity and collaboration in the classroom.

Thinking through these myths, confronting them deep inside ourselves, is the first step to freeing ourselves and our children from math anxiety. Try it, and I’ll be back with the next step next time.

Preventing Math Anxiety in Kids #2

Preventing Math Anxiety in Kids

A lot of what’s written about preventing math anxiety in kids is directed at classroom teachers. There are suggestions for creating an optimistic and safe classroom where questions are valued and group work is encouraged. But parents don’t run the classroom. So what can we do at home to help our kids grow up confident in math?

Raising math-confident kids starts early, and strategies evolve as the child grows. All along, though, the basic parental messages are the same.

• Numbers are cool. Math is fascinating, and it can be beautiful.
• People like you can do math. It’s a matter of effort and practice.
• People who use math well have power.
• It’s your right to get a good math education—but you have to work at it, too.
• Understanding math will give you lots of choices in your life and career.
• People learn in different ways; different topics are hard or easy for different people. The key is to find out what works for you.
• Always try to understand.

But parental messages aren’t just about what we say. They’re about what we do day-to-day. So what can we do concretely to send these messages about math?

Preschool and early grades

In young children, play games or share activities that include counting, measuring, sorting and comparing. Count out and sort the silverware or clean socks. Bake together or make up a mud pie recipe. Talk about more, bigger, heavier, less, fewer, smaller, and lighter. Notice and name shapes. Play simple board games with dice, or strategy games like checkers. Guess how many mailboxes are on your street and then count them. The main point is to have fun, and once in a while to mention that the fun things you’re doing are all a kind of math.

As children learn to write numerals, encourage them to be artistic if they choose. Let them write numbers in different colors or decorate their pages. Doing so won’t be helping them learn mathematical concepts, but it may increase their emotional connection and good feelings about numbers.

In the library, seek out storybooks that feature math. A few examples are Anno’s Math Games, by Mitsumasa Anno; The Doorbell Rang, by Pat Hutchins; Measuring Penny by Loreen Leedy; and One Hundred Hungry Ants by Elinor Pinczes.

Elementary school: the challenges of homework and math facts

As students enter elementary school, more of their math education is moving out of your hands. You’ll start seeing homework, and you may not be sure how to help. Start by expressing interest. Can your child explain what she’s learning? Can he show you how to do a certain kind of problem? For word problems, ask the child to challenge you by making up a similar problem of her own. Reason it through out loud. See if she can catch you in a mistake. Doing this accomplishes two things. Thinking backward about how to set up a problem requires real understanding and helps the child see how math connects to the real world. And hearing you reason, stumble, make mistakes and correct yourself helps the child see that mistakes are part of learning.

One big stumbling block for elementary school kids is mastering their math facts: addition, subtraction, multiplication and division. This is a tricky one. Many parents blame school curriculum if their kids have trouble mastering the multiplication tables, but the truth is that most kids struggle with multiplication no matter what math books they use. So how do we get our kids to practice to the point of fluency without convincing them that math is drudgery?

The key is to make practice a game. Multiplication bingo, computer games, rap songs--- do whatever it takes. A child who is fluent in basic facts has a much easier time of it in more complex mathematics.

When your child brings home a math test, don’t focus on the grade he received. Instead, look it over and comment on how much he’s learned. For missed questions, ask, “Do you understand what you did wrong?” Ask him to give an example of a similar problem and how he would solve it. Comment on how great it is that his positive attitude is helping him learn from mistakes

Middle School and Beyond

Older kids may not want you interfering in their homework. But you still have important messages to convey. Probably the most important (and one of the hardest to get across) is that the point of homework is not to rush through it as fast as possible. The point of homework is to understand the problems and the underlying mathematics. Kids should ask themselves what they’re trying to learn and check on whether they feel confident they have learned it. They should continue to make up questions for themselves. Encourage them to be proactive about asking questions of the teachers and other students. Tell them that questions show that they’re paying attention and thinking hard.

If your child is anxious about tests, strong preparation is key. Doing a little bit of review every day is more efficient than cramming on the last day before a test. Here the old trick of making up problems will come to your child’s aid. Study sessions with friends who ask each other questions are especially useful. We learn and retain more by actively coming up with questions and solutions than by reviewing old notes.

Encourage your child to knock off from studying early on the night before an exam. Taking a break can help her relax, but it also ingrains the idea, important for a healthy life in high school and college, that studying is something you pace over time.

If test anxiety remains a problem, your child should learn some relaxation techniques. Deep breathing exercises, visualizing success or a sunny beach, or silently repeating a phrase such as, “I am prepared and math makes sense to me” can all be helpful.

Finally, your child will be less anxious if he knows that no matter how he does on the test, you’ll still be interested, sympathetic, and supportive. Sure, the test is important, but bonds to parents are even more important. The math will always still be out there, ready to be learned.

Visiting Grinnell and grand challenges

I'm visiting Grinnell, Iowa, where my father grew up, and where my father, aunts and uncles, and many cousins attended college. Grinnell, a small college town with a lovely campus, is quiet and steamy in the summer. In the evening the cicadas are loud, and the last of the fireflies spark in the dusk. I'm staying in The Carriage House, a charming bed-and-breakfast on the English model.

I came for a summit of math and science teacher educators held by IMSEP, the Iowa Math and Science Education Partnership. I gave the opening address, entitled "Grand Challenges and Inspiration: Lighting the Fire in the Next Generation." In the speech, I argued that all our arguments about the need for more math and science education to preserve American competitiveness fail to inspire young people. I suggested that instead of haranguing them, we need to tap into their idealism and their desire to contribute as active agents in making a better world.

The National Academy of Engineering has articulated fourteen grand challenges (http://www.engineeringchallenges.org/) for the coming generation to meet, ranging from making solar energy economical to reverse engineering the brain and preventing nuclear terror. Not all of these are easily adaptable to the K-12 classroom, but some are--and probably more of them than I think.

I believe IMSEP will post the talk, and when they do I'll post a link to it.

I also ran a small breakout session asking whether it's feasible to mix math and literature at the middle school. The texts we looked at were Flatland, The Phantom Tollbooth, and my own upcoming book, Lost in Lexicon (See htttp://www.lostinlexicon.com). We discussed how Flatland investigates perspective, prejudice, and a limited viewpoint from both a social and mathematical perspective. We examined how an incorrect problem in The Phantom Tollbooth underlines the difference between reading a story and reading mathematics. Finally, I highlighted several examples from Lost in Lexicon of how mathematics and language intersect, as in a discussion of logic and irrationality, along with some interesting mathematical extensions in number systems, simple algebra, and plane geometry.

The session was an experiment for me, exploring how the mathematics of these books can be accessed at different levels of mathematical sophistication. It went well enough that I think I'll try to write up the talk to publish on my website or elsewhere.