(2023-04-04) Building Human Intelligence At Scale To Save The Next Generation From Chatgpt

Po-Shen Loh: Building Human Intelligence at Scale, to Save the Next Generation from ChatGPT. Our children's future employment prospects (and current educational experience, with ChatGPT-powered cheating) are in existential danger. There is an area close to mathematics, however, which devises solutions in which problems solve themselves even through self-serving human behavior: game theory. (this is a good peer-progressive story)

Intro: Po-Shen Loh is a math professor, researcher, and educator who transitioned to devise new solutions for large-scale real-world problems..... the real world, where he ultimately innovated fundamentally new approaches to pandemic control (https://novid.org) and scalable advanced live math education (https://live.poshenloh.com). He will also discuss educational strategies that build relevant skills to survive this new era of Generative AI (e.g. ChatGPT

Po-Shen Loh is a social entrepreneur and inventor working across the spectrum of mathematics, education, and healthcare, all around the world. He is a math professor at Carnegie Mellon University, and the national coach of the USA International Mathematical Olympiad team

last November-- all of us mathematicians were busy mocking it saying it can't do math

But it was clear that it could do some other things.

For example, it could write college admission essays

Please write a Yale admission essay. I like math. I also like computers, and I like hamburgers.. with great passion. Not really fancy big words, but sound like I really am passionate about helping people-- [LAUGHTER] --with hamburgers and math and computers.

It's not bad. It's not bad.

as far as I understand, one of the goals of the essay was to make it so people can't apply to that many colleges.

Actually, when I think about the purpose of each piece of the college admission, this helps to reduce the number of colleges people apply to.

But when you can generate the essay in five seconds, that's a different question

And then the next thing that hit me was that I was talking to somebody who worked in private equity.

So I was talking to this guy who now runs the company that his PE-- Private Equity-- firm bought. And so he's supposed to do something to fix the company. What do you think you're supposed to do to fix a company if you've bought one that is losing money? AUDIENCE: Lay off people.

So he had asked the marketing team, why don't you just use ChatGPT to write the blog post, and it was so much faster... And then the marketing team started to wonder about their jobs.

there was a Carnegie Mellon first-year student who came to chat with me about what he had done during winter break. And he told me he spent his winter break writing a program.

Tutor.com, by the way, is a website that you can use to get questions answered. You pay to get your question answered. And if you answer the question, you get paid

The questions and all the discussion are through chat only.

so when the question came in from tutor.com, he sent it to ChatGPT

So he knew, don't just ask ChatGPT to answer the question. He said, answer it as if you are a tutor

He took the response, and he passed it back into ChatGPT saying, please double-check.

if it failed the double-check, it texted him. So then he manually answered the question.

If it passed, then it went straight to the person. And he told me he made $600 before he got shut down

This thing shows your work. OK

I'll fool it, right? OK. All right. Now, what if you accidentally added to both sides of the equation instead? I was like this-- no way it can answer this. Oh, actually it answers like how I would talk. Wait a second. What about my job?

And then on March 14, there was GPT-4.

it tried to do the AP Chemistry Exam-- went from like about 20th percentile to about 70th percentile of humans.

But there are some other exams at the far end which are very, very low scores. Those at the far end are AMC 10 and AMC 12 and codeforces rating

the AMC are the American Math Competitions. That's how you eventually select for people to go onto the United States International Math Olympiad

The common theme of those questions is they're supposed to be different from the textbooks

And so you should not be surprised that they used to be at the very, very bottom

There was this strange phenomenon anomaly that on the AMC 12, it got to nearly 50th percentile

got me curious enough to start asking ChatGPT my own questions about math, which are harder.

This is something that you would learn if you went to high school algebra. (intermediate mathematics)

this should call into question what are we teaching our kids nowadays anyway?

The reason ChatGPT can do this is because it has ingested teacher's manuals.

after I saw these kinds of things where the tricks can be done in this way, it made me also realize we used to say that somebody is good at math if they somehow know a lot of tricks

This is the one-- I threw this on social media because this one blew my mind. Factorize 899

So is it a prime number? Ah, ah. It's not a prime number. By the way, normal people also make mistakes.

Find whole numbers x and y, so that 30x plus 31y plus 28 times 1 is 365.

There was a math contest I was at, and here's a question on the math contest.

so there are ways to do this. You can use modular arithmetic or whatever. Yeah, you know, let's go play with this thing. And it'll do it the brute force way-- brute force. This is, by the way, how I tried to solve it. When I saw this question at that math contest, I was on the screen, and I did this.

smart-- I don't use that word. I don't want to anthropomorphize. But it can do a lot of things, and that's why we need to think about being careful.

let's have it do something else. This is a math contest problem. So the numbers are special.

And I thought on Sunday this would break it, and I was wrong. What the heck?

If you go to really an average middle or high school student, they have trouble performing at this level.

how many people need to solve International Math Olympiad problems?

our collaborators in the Philippines. And they told us the sad news that very recently some of the large outsourcing companies were laying off 10,000 people.

There are populations in the world where vast numbers of people do text-based communications. Actually, even based on written chat. And if the company has the entire record of all written chats in the last two years in customer service, you could train a large language model on that entire body.

So I've started thinking about, what do we do with the next generation?

we might want to rethink on whether it's worth it to have people do this over and over and over again

there are people who are trying to integrate the language models with logic models to make it so they can even come up with new ideas

So what I'm trying to say is it is increasingly urgent that we do something to help the next generation to be able to come up with new ideas. That is, I think, one of the pieces that is going to be harder for these AIs to reach

I still teach at Carnegie Mellon

I just go in with a piece of chalk, and I say we got question or this theorem

Here's a theorem. We'd like to prove this theorem. Does anyone have ideas? And then we brainstorm.

And my job as the instructor is to use your ideas to solve the problem. This makes it more entertaining for me.

By the way, the proof they come up with is some wacky thing that I didn't expect would work, but it works.

And then I show, OK, now, another proof. And the student says, oh, I have this other idea. Now, we got another proof

I do the same thing with middle school students

if you're talking to students who aren't used to this, sometimes the students say, you're not doing your job. You're supposed to tell me how to do it.

this teaching style that I just described does not work in a general classroom. I know because I go to general classrooms.

It works at Carnegie Mellon University. It works in any gifted-- gifted is a bad word-- don't like that word. (TAG)

It works in any math club where people have all agreed they want to think.

And when you go into an average classroom, and you say, hey, everyone, let's brainstorm. 20% of the people brainstorm.

if you want to teach people how to think of a new idea, you could pull out people who actually are interested in trying to brainstorm and give them a chance to brainstorm together facilitated by someone who will show how their ideas will work.

This is called math team.

I was in Madison, Wisconsin-- public school. And it was the math team that saved me and discovered

And we weren't smart. We were just the ones who wanted to try harder, and someone came along and facilitated all of this.

I've come to the point where they have to come up with new ideas, and they also have to learn how to be as human as possible.

the interesting-- the most interesting and hard problems in this physical real world are the ones which are really fuzzy on what are the parameters you're trying to optimize anyway? (wicked)

And how do you balance this ethical thing versus that ethical thing?

And so in some sense, if you can learn to love and do well in the fuzzy problems, well, then, that's going to be something that will still be harder to automate

By the way, Khan Academy recently announced that they are working with OpenAI to attempt to make a totally automated tutor for teaching you the routine stuff.

But I actually came up with another way. It's actually very different. And the way that I approach the world is I think about the world through the lens of game theory.... somehow align all the incentives so that everyone does the right thing.

the age you really want to hit this on is younger, if possible, preferably sixth grade. (middle school)

So there was another problem I was thinking about. You see, I work with the country's top high school math students.

they have a problem, too.

They're trying to get into college.

It's actually usually around personality-- communication skills, to be specific... one of the biggest challenges to getting into a top university... their math is amazingly good-- amazingly good. And if they could lift their communication skills and interpersonal skills to the next level, they would be even more successful, not just in college, but in everything.

So I started a program for high school students. What we do is we bring in people who are really good at math. And then what we do is after we have these people, I hire professional comedians and actors and actresses to teach them how to be interesting. (interestingness)

So about five years ago, I was trying to get more people interested in math around the country

And the first thing the PR agency told me is that you have a big problem. You don't know how to talk to normal people. So they said you need to take performing arts classes. And so I did.

It was one of the best investments of my life. They only cost about $15 an hour. I'm strongly recommending these, by the way-- improvisational comedy classes. Improvisational comedy classes $15 an hour-- maybe because of inflation $20 now.

for an entire year, I was doing these classes for about three hours a week.

And by the time I was done, I never had to prepare for a single lecture again in my life.

This is what I discovered because it would give you the ability to use your domain knowledge to then dynamically run every single class, every single experience, to match whatever the audience has.

they have learned how to listen to people and read people and make people laugh at the right times and get their idea across

somebody came up to me afterwards and said, you are creating a generation of extremely dangerous people because they both can think super fast, and they could pitch any venture capitalist, build any team in any philanthropy they want and go.

That's why our first admissions point for this high school program is you have to be a nice person.

So we have this high school program. And, by the way, this works because it turns out that now everyone wins in this circle in the sense that the drama people-- we give them jobs

I found out that there were people who have great drama skills, and they're driving Uber Eats because you need to do something while you're auditioning.

we now have this ecosystem running on this side.

Suppose you actually want to learn how to communicate and to be able to talk and to make people-- you want feedback on your expressions. You need to talk. You can't just watch an expert.

What do you think people who are really, really good at math are happy to talk about? Math. Math. This now runs live video math classes.

Zoom classes-- because the only way to be able to do something like this at large scale across the world is to use the internet

you'll see two people teaching the Zoom class. The two people are high school students.

They are, by the way, both insanely good at math.

you'll see the interplay between two of the high school instructors

We are able to produce Zoom classes live at enormous scale across the entire world which all look and run like that,

any of you who have ever done math competitions know that you're used to looking at math competition questions with the first glance and having to think right away. And so what we did is we brought in people who are high schoolers who already have that skill.

well, there's one piece in this whole ecosystem that is the one that put the final incentive alignment inside.

what we have is inside every single one of these classes we have the basic unit, which is two high school students teaching, and we have a drama coach in every single Zoom class because there are enough performing arts people out there

Because now, here's why everyone wins. Middle school students-- they got this now.

Performing arts person is not a math expert at all. They are an emotional expert. They see the faces of the students learning. They see the faces of the people teaching. And they figure out how to match input and output to make everyone have a good time.

And then the other piece of incentive alignment is for the high school student. You know what you get?

we've actually given you the world's most efficient way to get one hour of coaching on your smile, and your delivery, and your posture, and your ability to magnetize. You put in one hour, you get one hour back.

Every high school student has actually a two-screen setup. We send them their equipment.

One screen they're able to see the image that they're sending through

And then they look and see what they're sending to the other-- the outside world. So these high school students, they can see the big merged image

The other screen, there is a feed directly from the actress or actor. Because the actor and actress-- actor or actress-- is inside the Zoom room with the camera off and microphone off. So the middle school students don't even know that she's there

So what we have here is the dram person is hidden except that there's a parallel video call going on on the other screen, where the drama person, their face, their smile, their gestures are visible, as well as their text-written chats.

These days, high school students are used to processing enormous amounts of streams of text coming at them.

So while they're talking, there is text comments coming in from the drama expert.

my University, CMU, we are very good at technology, but we also have one of the top drama schools in the country

By the way, the high schoolers, they have a track. The high schoolers-- they don't have to prepare for the class at all. This is a very important detail.

So all they do is five minutes before the class pop on the equipment, make sure everything works

their job is that they facilitate the discussion because remember, the whole point is the middle school students come up with ideas.

when I showed up. I'm not live.

When they're done they say, Professor Loh, how did you do the problem? OK. My video shows for three minutes.

Then she, the performing arts expert, unmutes entirely and talks in the audio channel giving all kinds of feedback on the last thing that just happened

An hour later, the high school students' job is done. We have made it so that the access time is approximately 10 minutes, and the one-hour dose in the middle is your access to a professional performing artist

we pay them, too.

I estimate approximately 100,000 high school students out there for whom this would be a very good thing for them to do. Maybe this won't help them all get into these dream schools, but it will give them some very valuable skill.

that also happens to be on the same order of magnitude as the number of people who participate in math competitions, like MathCounts, AMC.

The ratio is 10 to 1, by the way. 10 to 1 in the sense that for each high school we've got 10 middle school. So we could potentially-- this thing is something that could potentially scale to million.

I've done a bunch of things in my life, and most of them didn't work because that was before I realized we should make things that do what people want. (compelling)

It's a year later. It's a year later-- 100 high school students. We have over 1,000 middle school students who have been going through this thing.

where this fits in is that there are actually people, believe it or not, who are parents of middle school students looking for enrichment classes

The known market has a known price for the thing I just showed. And what we did is we undercut the price, and we increased the quality

I agree that an artificial intelligence could give all of that feedback and so on.

But if you notice what else we have told our high school students to do. It's to show love, to show care, to encourage, to make people really feel like being there

Actually, we always tell them, you're a game show host.

Why did I put two people there? I put two people teaching every class because I'm a graph theorist.

the concept is if you make the basic unit to always include two high school students, you make friends.

The other problem is there aren't enough spots in the Ivy League universities for all the students that we're working with.

And anyway, why do you have to go to Ivy League?

what you really want is you want to get into some ecosystem where there are other people who trust you, who know you well enough, who at some point when you want a job will recommend you. (job market)

if you spend a few hours co-teaching with somebody, and you build a graph. (social graph)

When you teach for 20 hours with somebody else you are not competing with them.

But we have a heart. So we actually also care about the underserved. There's a whole other arm of this, where there are students who don't have parents who are enrolling them in this. So we've gone and talked to philanthropists.

specifically recruit classes of high motivation, low access students.

I wanted to leave enough time for free-ranging Q&A because that's way more fun.

The one missing the element from your story is stories.

our high school students do use that, too.

maybe the stories will be about some cool math thing or some cool experience they had with math.

This is actually why we give them the improvisational comedy training. Improvisational comedy sets you up

to randomly tell a story at any given time, and then pull back off the track

they just know that there are these particular videos which will get you back on track, and they have the time points to do that. In between, they can carry between by either using the ideas students have, or if they see the need to do the story thing,

we get office hours with you every single week. And whenever we do the office hours we always go around, and every high school student shares a tip that they were thinking about this week about communication or performance or teaching

how we are going to protect against our speakers.

That's why we don't take any investors. (OPM)

I've just showed you a cash flow positive, bootstrappable ecosystem, where you get paid before the [INAUDIBLE]. This is called exponential growth.

And I don't want somebody who will be on the board of directors saying, I have a way of saving 1% of your cost by doing x.

whenever a piece of technology is made or there's a company, I always am curious what is the purpose of the company?

as we look at the various powerful companies in the world, they are not actually necessarily entirely run by their founders

you see the incentives don't quite align because sometimes there might be an idealistic person who does not control the board of directors

what do I think about ChatGPT though? I think whether it's being intentionally used for evil things

And so what are we going to do about that? We're got to think.

I just mentioned philanthropy because we've got two arms. We've got one arm, which is just purely growing this thing and helping to help people learn. But I also have actually spent a lot of my time trying to make sure that this new thing doesn't exponentially expand the equity gap because I also work in education.

How many people are trying to get into a computer science major are actually interested in CS? How many people trying to get into Haas are interested in being an entrepreneur? Not many. Maybe 75% of people are trying to be a consultant and earn some money and call it a day.

what's your perspective on this, like how your education methods can actually let people have passion, not just let people adhere to the determined qualities that society wants us to have, but...

I tell all of my students is that the way you succeed is not by getting some famous degree. It's by making sure you have lots of really good friends who are capable and can go to powerful whatever who are two years older than you because then you get a job through them. networking

What I realized is that you don't have to build your network out of small, complete graphs. (weak ties)

What I said is you can also have an M vertex graph for every vertex. That's degree 6. You can find a way to just make it so that all these people from 14 years until 18 years have co-taught and built strong relationships with 6 to 10 other people.

I'm just building a random network where every node has 6 to 10 connections, which are good, with other people who have been selected because they were nice. They were very good at math. And somehow, they were pretty committed to thinking that interpersonal skills are important. These are the three criteria to get into our high school program.

And so what we did is we said, look, let's just build this network. Suddenly, it doesn't have to be limited Ivy League's walls.

And also, by the way, that's also why we run our weekly meetings. It's all about these other things we're talking about, like the passion, having some reason for doing something.

it's very good if you can move away from the attitude of, how do I get stuff for myself, and move into the attitude of how do I create value with my situation I am in?

if you just keep running around, say, how do I make value out of this situation by bringing together different people, by adding a new idea, by adding something I know, just add value. Don't worry about what comes back to

I run around the country telling people about this kind of philosophy. So I'm basically trying to change this philosophy on a national scale. And I realized the easiest way to do that was to show up in the city and declare a talk.

what happens to high school now when like ChatGPT can do high schools?

High school just got completely broken.

She said, kids are using this to write their essays. She was asking me, can you make a detector? I said, good luck.

the solution that most people have seemed to find to this is called do homework-- no-- do evaluations in class, so write essays in class.

I was talking to seventh graders. I said ChatGPT. And they said, we hate ChatGPT. I said, why? Everyone else thinks is awesome. They said, because now we have to write essays with pencil.

is it still worth teaching it?

should we change all of the curriculum?

It's very difficult to do something at that scale

I have come to the conclusion that I as a single person will not be able to convince powerful people who control curriculum to change curriculum.

So what I do is I make an out-of-band solution that is just better. (route-around)

And that's why we have the researchers. Because the point is once you have the researchers documenting it, then suddenly there might be a question

How do you assess if a given high school student is nice? We take advantage of the fact that most high school students haven't learned to be sophisticated liars yet.They get to the interview. The interview is with a couple of people-- myself, somebody else is a full time, and then also some high school students who are actually inside the program

We don't say, do you care about people? You can actually tell. And I want to emphasize, we take people who are quiet, too.

before they even get into the teaching part of the program, they have to go and practice for a while.

Paid class means delivering the goods. We don't let them do that until they're checked out on being able to do what you just saw here.

this idea, by the way, doesn't have to only teach math. It could generalize to all things. In that sense, I'm not a very good business person.