Sam Altman - Lecture 16: How to Run a User Interview lyrics

This text is annotated! Click on the highlights to read what others are saying. If you'd like to add your own insights, comments, or questions to specific parts of the lecture, visit the lecture page on Genius, highlight the relevant text, and click the bu*ton that pops up. Your annotation will appear both here and on Genius. Sam Altman: Good afternoon. Today's guest speaker is Emmett Shear. Emmett is the CEO of Twitch, which was acquired by Amazon, where he now works. Emmett is going to talk about how to do great user interviews; this is the talking to users part of "How to Start a Start Up." It should be really useful. Thank you very much for coming! Emmett Shear: Thanks Sam. I started my first startup with Justin Kan right out of college. We started a company called Kiko Calendar. It didn't go so well. Well, it went alright. We built it, we sold it on eBay. That's not necessarily the end you want for your start up. It was a good time. We learned a lot. We learned a lot about programming. We didn't know anything about calendars. Neither one of us were users of calendars. Nor did we, during the period of time we did the thing, go talk to anyone who actually did use a calendar. That was not optimal. We got the build stuff part of the startup down. We did not get to the talk to users part. The second startup we started, we used a very common trick that lets you get away with not talking to users, which was that we were our own consumer. We had this idea for a television show, Justin.tv, a reality show of Justin Kan's life. We built technology and a website around the reality show we wanted to run. We were the users for that product. One way to cheat and get away with not talking to many other users is to build something that literally is just for you. Then you don't need to talk to anyone else because you know what you want and what you need. But that is limiting way to start a startup. Most startups are not built for the person who is using them. When you do that, every now and then you get really lucky and you are a representative of some huge cla** of people who all want the exact same thing you do. But very often, it just turns into a side project that doesn't go anywhere. We kept working on Justin.tv for awhile and we actually achieved a good deal of success because it turned out that there were people out there who wanted to do the same thing we did, which was broadcast our lives on the internet. The issue with Justin.tv that kept us from achieving greatness is we hadn't figured out how to build towards anything beyond that initial TV show. We built a great product. If you wanted to run a live 24/7 Reality television show about your life, we had the website for you. We had exactly what you needed but if we wanted to go do more than that. We wanted to open it up to a broader spectrum of people and use cases, but we didn't have the insight to figure that out because we weren't that user. So we decided to pivot Justin.tv. We decided we needed to go in a new direction. We thought we built a lot of valuable technology but we hadn't identified a use case that would let it get really big. There were two directions that seemed promising. One was mobile and one was gaming. I lead the gaming initiative inside of the company. What we did with gaming that was very, very different from what we'd ever done before was we actually went and talked to users. Because while I loved watching gaming videos, I was very aware that neither I nor anyone else in the company knew anything about broadcasting video games. I was amped about the content. I thought that there was market there. That was the insight that wasn't common at the time, which was how much fun it was to watch video games. Quick show of hands, how many people know about watching video games on the internet here? If you don't know about watching video games internet you should go read about it, because it's important context for the stuff I am going to talk about. The main point is I thought it was awesome, but I didn't know anything about the important side of it, which is actually acquiring the content of the startup broadcasting. We ran a very large number of user interviews. We talked to a lot of people and that data formed the core of all the decision making for the next three years of product features on Twitch. We continued to talk to users and in fact built an entire part of the company whose job it is to talk to our users. That is a whole division we didn't even have at Justin.tv. We had no one at the company whose job it was to talk to our most important users. I want to give you guys a little bit of insight into what it meant to talk to users. We determined that the broadcasters were the most important people because when we went and looked into the market, we looked into what determined why people watched a certain stream or went to a certain website. They would just follow the content. You had a piece of content you loved and the broadcaster would come with you. That's actually the one really important point about user interviews, which is that who you talk to is as important as what questions you ask and what you pull away from it. Because if you go and talk to a set of users, if we had gone and talked to viewers only, we would have gotten completely different feedback than if we were talking to the broadcasters. Talking to the broadcasters gave us insight into how to build something for them. That turned out to be strategically correct. I wish I could tell you the recipe for figuring who the target user is for your product, and who your target user should be, but there isn't a recipe. It comes down to thinking really hard and using your judgment to figure out who you are really building this for. I want to do something interactive now. I'm going to get a bunch of ideas from you guys and I'm going to pick one of them. I want everyone to sit down and do step one of this process right now. Which people, where would you go to find the people you needed to talk to in order to learn about what you should build. The idea we are going to use is a lecture focused note taking app. The idea is: the state of the art for note taking is not good enough yet and I want to make a note taking app that improves that experience. It will make taking notes in cla** better. Maybe it has collaboration features or maybe it helps you focus better somehow. It has multimedia enhancements. All sorts of possible features. That's the idea. So take 120 seconds right now and think about not what you would ask or what the right features for this app is, but who would you talk to? Who is going to give you that feedback that is going to tell you whether this is good or not. It's good to think in your head but actually write it down and come up with the five people you would talk to. The five types of people you talk to, and who you think the most important one was. There's nothing like actually running through something and trying to do it. Actually get it into your head that it's right way to do it. I'm gratified to hear the clicking of keyboards now. If you are following along at home actually do it. Think about who would you talk to because that's the first question for almost any startup. You need the answer to the question: who is my user and where am I going to find them? Alright, that's shorter than you normally would think about this problem. It's actually a really tricky problem, figuring out where to source people is pretty hard. We're going to move along in this highly abbreviated version of learning how to build a product and running a user interview. Can I get one volunteer from the audience to come up and tell us who you would talk to. And we'll talk about it. Audience Member 1: I would definitely talk to college students first, because we sit in a lot of lectures. Specifically, I want to talk to college students studying different subjects to see if they are an English major, if that makes a difference versus studying Math or Computer Science in terms of how you want to take notes in different lectures. Emmett Shear: You're going to talk to a bunch of college students. Would you pick any particular subset of college students? We don't want to talk to all college students. Audience Member 1: I want to only talk to college students and break down the divisions by people who study different areas. It would make sense for people who have different study techniques, because some people take a lot of notes. Some people don't take that many notes but still jot stuff down. Emmett Shear: That's a really good start. Those are obviously the users you want to go talk to, especially if you are targeting college students as the consumer. If you are talking to college students as the consumer, you are going to get a lot out of students about what their current note taking habits are and what they would be excited about. One of the problems with selling things to college students is that college students actually don't spend very much money. It's really hard to get you guys to open your wallets, especially if you want them to pay for a school related thing. People don't even want to buy text books. You probably all use checks or debit or borrow from your friends. So one of the things that I think you would be missing if you go after just the students, is who the most important person to this this app is. If you actually had a note taking app for colleges, the people most likely to actually buy a note taking app would be college IT. Presumably for the most part you want to sell software to students, and the people who have to be brought into that is usually the school administrators. That might be one approach. I feel like you will presumably go talk to college students and find out they don't actually buy any note software now at all. It's possible they do, in which case I'm completely wrong. This is why you have to go and talk to the users. But you then have to try other groups. So I would talk to college IT administrators as well.That's another area that's really promising. You might talk to parents. Who spends money on their kids' education? Who is willing to pull their wallet out? The parents of kids who are freshmen going to college for the first time. You need this app to make your kid productive so that they don't fail out of college. There are actually a lot of groups that aren't necessarily obvious users but who are potentially critical to your app's success. When you are at the very beginning of a startup, when you have this idea that you think is awesome, you want to have the broadest group you possibly can. You don't just want to talk to one type of person. You want to get familiar with the various kind of people who could be contributing. Let's have someone come up and we're going to pretend we are running a user interview. We are going to talk to a college student and try to find out what we should build, what we should get into this note taking app. Another volunteer please, for running an interview. Hello. Audience Member 2: Hi, I'm Stephanie. Emmett Shear: Hi Stephanie. Audience Member 2: Nice to meet you. Emmett Shear: Welcome. Thank you for agreeing to do this user interview with us. I want to hear from you about your note taking habits. How do you take notes today? Audience Member 2: Sure, I take notes in a variety of ways. Because of speed and efficiency, and because I can come back to it later, it's easy for me to take notes on my laptop. A lot of those would be primarily text based, but in certain cla**es, for example if I am taking a History cla**, most of it would be in text. But if I am taking a Physics cla**, there are going to be more complex diagrams and different angles I have to draw. Emmett Shear: What software do you use for this stuff today? Audience Member 2: I just do pen and paper for that. Emmett Shear: You do pen and paper. So you do a combination. You take notes with pen and paper. You take notes with the computer sometimes. Audience Member 2: Yup. Emmett Shear: When you take all these notes, at the end, do you actually review them? Be honest! Do you actually go back and look at this notes? Audience Member 2: The pen and paper not so much. But yes to the software based. It's easier to access and it's easier for me to share and collaborate and maybe even merge notes with cla**mates and friends. Emmett Shear: What did you use to take notes today on your computer? Audience Member 2: Google Docs and Evernote. Emmett Shear: Why two things at the same time? Audience Member 2: Evernote is easy if I am trying to just collect for myself. You can share, but Google Docs is easier to share. If a friend has already created a folder in Google Docs, I just have to add to that folder. If it's for my personal use I tend to go more toward Evernote. Emmett Shear: It sounds like you have a lot of note taking collaborations. Audience Member 2: Yeah, I wish it was integrated. Emmett Shear: Tell me more about that. Do you wind up taking most of the notes, most of the value out of notes that other people take? Or is it mostly your own notes you review at the end of the semester? How does that work? Audience Member 2: It's mostly mine because I am pretty picky about the way I like things organized. Design wise or formatting, even color, I am really particular with. The font that we use really effects the way I study. So I tend to personalize notes, even after I merge them. Emmett Shear: So you're pulling notes from other people but then you merge them into what works for you. Awesome! If you have Evernote notes and you have Google Docs notes, and you have pen and paper notes, once the semester is over, do you ever go back to any of that stuff or is it per quarter? Once the quarter is over, do you ever go back to any of that stuff? Audience Member 2: For cla**es not so much, but if it's notes that I have taken for talks, like these for example, or if it's interview prep, I tend to go back because I like to keep these things fresh in my mind. They help me prep for future things. Emmett Shear: That's interesting. Tell me more about that. You take notes not just in cla**. Audience Member 2: I take notes to summarize main points. For example, inspirational quotes from talks like these. If I am going to an event where I am going to meet someone, notes help me remember what was at the talk. Emmett Shear: Awesome. Normally I would actually dig into a lot more detail. There are a huge amount of open questions that are still in my mind after hearing that. Which people do you collaborate with? How long are your notes?. How much time do you spend note taking? I would dig into her current behavior but in the interest of time and not making everyone hear about the intricacies of one person's note taking habits forever, we're going to move on. Thank you very much Stephanie. Audience Member 2: Thank you. Emmett Shear: I appreciate that. You notice we are not talking about the actual content of the app at all. I'm not really interested in features. I don't want to know about a specific feature set in Google Docs or Evernote. I might dig a little more into which features actually get used. If she's actively collaborating, how does that work? I heard some interesting things, " We use a folder." That's interesting to me. The main thing you're trying to do when running this first set of interviews is not necessarily ask questions about optimizing user flow. Or questions about the specifics of any of that stuff. That can be distracting because users think they know what they want. You get the horseless carriage effect where you're asked for a faster horse instead of asked to design the actual solution to the problem. So you want to stay as far away from features as possible because the things they tell you feel overwhelmingly real. When you have a real user asking you for a feature, it's very hard to say no to them because here's a real person who really has this problem. They're saying, "Build me this feature." But as you start to talk to lots of people and really get a sense for what their problems are, you figure out if this is actually a promising area or not. Based on what I heard there, starting from that user interview, I'm not necessarily positive there is a problem. At least there might not be a big enough problem that it's worth building a whole new product for. I didn't hear a lot of things that were big blockers, where there is something really wrong with the way it was working. Unless I have some big idea, I would take that as a negative sign. That doesn't necessarily mean you can't move forward and keep talking to more people. Just because you don't get anything out of talking to the first person doesn't mean there are not going to be more people who actually have a problem. Once you've talked to about six to eight people, you are usually about done. It's unlikely you're going to discover a bunch of new information. Which is why it is important to talk to different extremes of people. Go find people who are different, because if you talk to six Stanford College students you are going to get a very different response than if you talk to six high school students or six parents. Based on that though, you can come up with a set of ideas.You have this information about how someone takes notes. You had some ideas as to how you could build something cool. If you are going to build just one feature on top of Google Docs, what would that feature be? For a new product like this, it might be a good way to start thinking about where to go. They are extensively using this thing right now, how can we make that experience just one quantum better? Something that would be really exciting to this person, something that would be one step ahead. Take two minutes and think about what that feature might be. Try to come up with what you might do based on what you heard from Stephanie that could convince her to switch away from her current collaborative, multi-person, all working together work flow on Google Docs to your new thing that has the features of Google Docs plus this one special thing that is going to make it more useful and convince them to stop using the thing they are already using. Awesome, alright. I am going to invite our third guest up. Audience Member 3: The reason she uses Evernote is because of sticky note type notes. More thoughts and like details. I feel that Google Docs has documents and not smaller notes. I feel like a feature that would be super would be a mobile version of drive that isn't clunky and doesn't make you use documents could be useful. Emmett Shear: Awesome. That's a good insight. That's exactly one of the things you get out of that user interview. Now you have this idea. You've gotten this user's feedback. What if we had a Google Docs that had the collaborative aspects and the group aspects but where you could pull in more little one off notes. A product designed more around note taking. The question is now, once you have this idea, is this enough? Is this something people would actually switch to? There are two ways to validate that. One, if you are quick at programming you can literally just go build it, throw it out into the world, and see what happens. When that works, it's an excellent way to approach the problem. But a lot of the time that one little thing that's a bit better might take you three months to actually build. So you want to go out and validate that idea further before you start building it. You might take that idea and draw diagrams of what it would look like. Draw the work flow and put that in front of people. The one thing you really don't want to do is ask them about a great idea for a feature. Ask them, "Are you excited about it?" Because the feedback you get from users if you tell them about a feature and ask them, "Is this feature good?" is often, "Oh yeah that's great." When you actually build it, you find out that while they thought it was a clever idea, no one actually cares to switch and get it. So the one question you can't ask is, "Is this feature actually good or not?" Sam Altman: What is the minimum that you can do to actually build on that question? Between asking and actually building the full thing? Emmett Shear: What's the minimum you can actually get away with to validate your product, given you can't actually just go and ask them, "Is this good or not?" It's highly dependent on the particular feature. Usually the best thing you can do is just hack something together. If your idea is to build something on top of Google Docs, don't go rebuild Google Documents but for note taking. Find a way to write a browser extension that stuffs that little bit of incremental feature in and see if it's actually useful for people. Find a way to cheat is what it comes down to, because if you can't actually put it in front of people it's really, really hard to find that out. For bigger things, where you are actually trying to get people to spend money, it gets a lot easier. If you are selling, it's great. Sales is the cure-all for this problem. Get people to give you their credit card and I guarantee you they are actually interested in the feature. It's one of the most validating things that you can do for a product. Go out there and actually get customers to commit to pay you up front. The problem when you are working on a student note taking app, is that's going to be relatively hard. Unless your idea is that you're actually going to sell it, the trial version is probably free. You're not necessarily going to learn that much by trying to charge people money. But if you go out there and can get people to say "Hey, I am going to give you money," the money test is amazing. It clarifies whether or not they're really excited about your product. If you're not five dollars excited about it, you're probably not very excited about it. The last thing I want to do is work through what happened at Twitch. I brought some slides that I'd like to put up. They are representative excerpts of Twitch feedback. I had a twenty-six page document of all the feedback and realized that reading that was going to be a little bit tedious. Lots of people said this to us when we asked them questions. I've pre-condensed the feedback for you. To launch Twitch, we talked to a bunch of existing Justin.tv broadcasters and asked them about their experience broadcasting, what they liked about broadcasting, why they broadcasted, what they broadcasted. What else was going on in their life? When you talk to detailed users of your product, they come back to you with very detailed things about features because they get mired in the features. You have to sort of read between the lines. They ask us for things like, “I want a way to clear the ban list in my chatroom." That was actually a very common request because there was a very particular issue with how our chatroom is worked. People would ask for the ability to edit titles of highlights after creating them. This stuff was really consistent. As we talked to broadcasters, we probably talked to twelve to fourteen broadcasters of the Justin.tv gaming platform, we got all this feedback. "Your competitors have all these cool features like polls and scrolling text. I can personalize chat there." Then we have some positive feedback. "You guys don't have ads. You're able to ban trolls." A bunch of stuff about chat, around interactivity with their viewers. That was all really interesting. This is what the Justin.tv broadcasters wanted us to build. This is where they felt pain using the product. If you thought that what we did was go and address these problems, you would be wrong. People who are using your service already are willing to put up with all these issues, which kind of means that these are probably not the biggest problems. If you are willing to ignore the fact that you can edit the ban list and that titles are editable, that there is no way to get trolls out of your channel and you're using the service anyways, maybe those aren't huge problems. That brings up a really important point, which is you have to compare groups of people. And compare the level in which they argue. Here we have competitor broadcaster feedback, which is really interesting. This is stuff we heard a lot from people who were using other broadcast platforms. They wanted to be able to switch multiple people onto their channel at the same time. They complained about us not having a revshare program. They talked a lot about how they were trying to make a living and they really wanted to make money pursuing this this gaming broadcasting thing. They talked about video stability. Our service wasn't good in Europe. Globally, video stability was this huge, huge issue. If you compare and contrast, it was really different. What people who didn't use our service cared about was completely different than the people who were using our service. We focused on this stuff because this was the stuff that was so bad that people weren't even willing to use our service. Most of it we hadn't actuallythought about it because our user base happened to be well educated and knew about all their options. Reaching out to them meant that they probably already tried all four services and actually had an opinion. It's great when you can get users who are that informed and understand the space that well. The other important thing we did was talk to non-broadcasters. We talked to all the people who weren't using us or our competitors. In many ways, those were the most important people. Talking to your competitors is a a short term win, unless your software is like Google, which is a search engine which everyone uses, then there may be no non-users to convert. In the case of gaming broadcasting, almost everyone is a non-user. The majority of people you are competing with are non-users. They are people who have never used your service before and what they say is actually the most important. What they say is the thing that blocks you from expanding the size of the market with your features. If all you do is look at your competitors and talk to people who use your competitors' products, you can never expand. You're not learning things that help you expand the size of the market. You want to talk to people who aren't even trying to use these things yet. Who've thought about it maybe, but who aren't into it. What did they say? My computer isn't fast enough. I am focused on training twelve hours day for the next tournament. I like making the perfect video and I like editing it. I upload a couple of things to YouTube. I don't do live streaming. I have no desire to go into that space. In Korea this is a big problem. Once our strategy gets broadcast in major tournaments, we have to start over. We have to come up with an entirely new strategy. The last thing we ever want to do is broadcast our practice sessions, are you crazy? That's going to hurt us in the next big tournament. This became big outreach program for us, trying to figure out how to get people over this. We brought people computers. We worked closely with gaming broadcast software companies to help the people who made the broadcasting software. We started building broadcasting into games and into platforms. We built broadcasting into the Xbox. We brought broadcasting into PlayStation 4 because we needed to overcome this issue. Broadcasting wasn't possible. These were the three big groups we looked at for broadcasting. You combine that feedback and what it tells you is not features to build, because the features they asked for, polls, the ability to have a child account, we haven't built most of that stuff. What was important were the issues, the goals they were trying to accomplish. People wanted money. People wanted stability and quality. People wanted universal access for viewers all around the world. That became our focus. We dumped almost all of our resources into things no one ever mentioned in an interview. Those were the things that actually addressed the problem. The way that you can tell that it worked is as we would build these things, we would go back to exact same people we interviewed and we would say, "You told us you really cared a lot about making money. We built you this subscription program that will let you make money." It's astonishing because most people had actually never had that experience. They had never talked to someone and said, "It would be really great if your product had feature X" and then a month later your product actually has feature X, or at the very least a feature that addresses the problem that they brought up. The people we converted first to our product were the people that we talked to about user research. They were the ones who were the most impressed. Which is fun. It really worked, because we picked people who were representative. We picked big broadcasters. Small ones. Medium ones. We made sure were addressing their concerns. That was completely different from how we approached the problem at Justin.tv. With Justin.tv when we tried to do this, we'd go through huge amounts of data. We spent tons of time looking through Google an*lytics, Mixpanel, and in-house an*lytics tools. Figuring out how people were trying to use the service, where our traffic came from, completion rates on flows. You can learn things from that. I'm not telling you not to look at your data. But it doesn't tell you what the problems are you need to address. We would invent these ideas at Justin.tv without talking to someone and then nine times out of ten, that idea would turn out to be bad. That's actually one of the disappointing things about doing user interviews and getting user feedback, which is why I think so many people don't do it. You're going to get negative news about your favorite feature most of the time. You're going to have this great idea and you're going to talk to users and it's going to turn out that nobody actually wants it. They are actually concerned about a completely different set of things and they don't care about what you thought was important at all. That's a little bit sad, but think about how sad you'd be in four months when you launch that feature and it turns out no one actually wants to use it. That's the lecture section. I want to take some questions from the audience. Audience Member 4 : What do you see startups get most wrong about interviews? Most startups don't do them at all, but the ones that do, what are their most common mistakes? Emmett Shear: The most common mistake is showing people your product. Don't show them your product. It's like telling them about a feature. You want to learn what's already in their heads. You want avoid putting things there. The other thing is asking about your pet feature direction. If you think you want to add subscriptions to your product, going and asking people, "Would you pay for a subscription? Would you use this feature?" Another big mistake people make is talking to who is available rather than talking to who they need to talk to. There are certain users are really easy to get at because they are members of your forum already. You have some product forum, you talk to the users on that forum because they're easy to access. We spent weeks digging for identifying information and figuring out who these people were so we could talk to them. This was a site that did not support messaging, so there was no obvious way to interact with them. We spent a bunch of time trying to network and find those users. Because if you just talk to who's easy to talk to, you're not getting the best data. The fortunate side there is almost everyone is flattered to be asked what they think, so they will actually talk to you and tell you things. Audience Member 5: How hard was it to get buy in from the rest of your company? You can say, "I'm in charge so you're doing what I say" but that's probably not the best way of doing it. Emmett Shear: That's a good question. If you just go to them and say, "I talked to the users. I figured it out. We have to build this," it's really hard because people don't trust you. There's something magic about showing them the interview though. I recommend recording interviews. It also stops you from taking notes in the middle, which is a little bit disruptive. It makes it hard for you to actually engage in the conversation. You can then play that recording for people. They don't have to be there for the entirety of all the interviews, but when you want to make a point about what what you should be building and why, you can play the interview back for the rest of the company. It's like magic, the influence it has on people's thoughts, on what is the right thing to build. Audience Member 6: Since you mentioned recording, did you try to insist on doing Skype interviews rather than over email? Emmett Shear: You definitely want to Skype. You don't want to do interviews over email if you can avoid it, because interviews over email are non-interactive. The most interesting learnings come from the, “Interesting. Tell me more." The instant you hit this vein, they will say something that you didn't expect. And then you should drop into detective mode. Detective mode is, "Huh, that's interesting. Can you tell me more about that?" People don't like silence, so they'll keep talking to feel the void. The best part about doing an interview over Skype or doing it in person is that you have that interactive feedback. You can actually pull a lot more out of people. Email interviews are basically useless. In person or over Skype interviews are also easy to record. Make sure you ask them if it's ok to record. It's not polite to record people without their consent, but if they are willing to give you a user interview, they'll probably willing for you to record it as well. Audience Member 6: What about the international market? You mentioned that you have a lot of users in Korea. Emmett Shear: That's really hard. To this day Twitch works way better in English speaking countries than it does in non-English speaking countries. A big part of that is we are much better at talking to people in English speaking countries and learning what their needs are. We are not as good at that in other Countries. We've tried to address that by hiring people who speak Korean and having them translate. We've tried to address it by finding representative people who speak both English and Korean and reaching out to them. But the problem is you're not actually getting a representative sample, no matter how hard you try. The very fact that they are a fluent English speaker means they are not representative of all the people who don't speak fluent English. It's a hard problem. It's why companies find it easier to build markets in their home country. It's really hard to talk to users abroad. Audience Member 7: What channels do you use to reach out to them? And do you ever compensate them? Emmett Shear: The channels we used to reach out to them were onsite messaging systems. Most websites have some way to contact the user. If they are a visible user of another website, you use that site's messaging system and say, "Hey. I was watching your stream...” Or, "I'd love to ask you some questions about your usage. Would you mind hopping on a Skype call?" We also find out where those people were. We'd run into them at events because a lot of these people go to the same events. We wouldn't run the user interview at the event, but we'd get to know them. We would exchange business cards and we would get in touch with them. We tend not to compensate people. If people don't care enough about the problem to like someone who is trying to solve it, you're probably barking up the wrong tree. We never had any trouble getting people to talk to us without paying them. Audience Member 7: What about onsite user feedback tools? Emmett Shear: This is a whole second set of user feedback that's really important. You're talking about when you have a new product and you want to see if it's actually going to work or not. You put it in front of people and see how they use it or not. That is super, super important. It can tell you where you went wrong building something before you launch it, which is great. It doesn't tell you what to build. It helps you iron out the kinks and edges of the thing you did build. Generally speaking, that wasn't the user feedback we were getting. I mean that stuff's good, it's much more similar though to the data driving approach. You're finding out why people are dropping off of this flow. You're not finding out the problem you should really be solving. What do they care about as a human? This early stage user interview is crucial for startups. That's where you want to focus. We didn't bring anyone in onsite, it was almost all over phone or Skype. Audience Member 8: In finding groups of people that can give different kinds of feedback, is there a group that you should focus on first? Emmett Shear: Given that we had very limited resources, we focused on the people using competing products. We knew that they were already interested in the behavior that we needed and they were willing to do it at all. Therefore all we had to do was convince them to switch, which is much easier to do than to create a new behavior. We did that because we had to get some quick wins. My gaming project inside of Justin.tv would have been k**ed if wasn't showing twenty-five percent month over month growth every single month. That meant focusing on the short-term, on getting the people in right now. That turned out to be good in general. Audience Member 9: In the beginning the video gaming industry was decentralized. There wasn't a lot of cohesion, but now it's very different. You said you originally spoke to broadcasters and streamers themselves. How has that changed? For example, ? has banned users or professional players from streaming their own stuff. Did you try to gain leverage with that? Emmett Shear: Yeah, so the question is about the game publishers. Game publishers are important people in the space. Any big company for that matter isn't going to give you the time of day as a small startup. Which is both good and bad. It means you don't need to talk to them because they're not interested in you. But it means you actually just can't talk to them. We tried but no one wanted to talk to us. They did once we started getting some traction and becoming a bit of a player in the space. I don't really want to talk that bad about them because they were nice enough about it. When you are a tiny little startup, there are lots of tiny little startups, and they don't have the time to talk to all of you. As we've gotten bigger, game publishers have become increasingly important for us. If I was to talk about who Twitch does user interviews with now, who we pull information form now, it would include game publishers. Definitely! They've become much more active in the space. They weren't particularly active three or four years ago. The really important user interviews in general are from the pool of people you care about, and that is going to shift over time. The people who get you started for the first six months are not who will be using it three years later. It's very important you keep doing this stuff. One of the things that is really easy to do is to do a little bit of it in the beginning and achieve some success and then stop talking to new people. That's a good way to make the next set of features you build be not as good as the first ones. Audience Member 10: How do you give good user feedback if you're a user? Emmett Shear: How do you give good user feedback? I want a user to tell me what they are really thinking. What their problems really are. To just sort of ramble. I want someone to just tell me about stuff in their life. The more you learn about them as a person and the context of what they are doing, the easier it is to understand why they want the things they want. That's really the critical question. What I am looking for in someone when I am doing a user interview is someone who is going to be willing to talk a lot and be willing to give me a full picture. On flip side, if you want to help people out with good user interview feedback, ramble.Just talk about everything. Alright great. Well thank you very much! Sam Altman: Thank you very much!