April 18, 2023

The Risa Stein Hypothesis: Trust is a Crucial Component of a Successful Product

In this episode of the Product Science Podcast, we cover what it means to be a product manager of Integrity, what key areas are involved, and how Risa works with PMs across Slack.

The Risa Stein Hypothesis: Trust is a Crucial Component of a Successful Product
Written By:
Holly Hester-Reilly
Holly Hester-Reilly

Risa Stein is a Director of Product, Integrity leading teams focused on protecting customer safety, privacy, and security. Risa and her team are responsible for setting and enforcing policy, investigating and stopping bad actors who try to abuse Slack, and developing new product features to protect Slack's customers. Prior to joining Slack, she led Transparency and Safety Experience products at LinkedIn and worked in Trust & Safety and product at Twitter. Risa received her JD from Stanford Law School, an MBA from the Stanford Graduate School of Business, and her honors BA from Brown University. Risa lives in her hometown of San Francisco with her husband and two dogs, Noodle and Pancakes.

In this episode of the Product Science Podcast, we cover what it means to be a product manager of Integrity, what key areas are involved, and how Risa works with PMs across Slack.

Subscribe for the full episode on Apple, Google Play, Spotify, Stitcher, YouTube and more. Love what you hear? Leave us a review, it means a lot.

Resource links

Follow Slack on Twitter

Follow Risa on LinkedIn

Follow Holly on Twitter

Follow Holly on LinkedIn

Questions we explore in this episode

What inspired Risa to work towards making the internet a safer and more equitable place through product management?

  • Risa went to law school with the desire to work towards fairness and equity
  • Risa also got an MBA, during which many of her peers were interested in product management
  • Risa realized that she could use skills she learned at both law and business school such as persuasive communication and advocacy in a role at the intersection of product and policy

What are the three broad areas Risa's role as director of product integrity at Slack entails?

  • Integrity product: This involves ensuring that Slack's products are designed and built to foster trust and safety for all of its users. Risa's team works on developing features that help users manage their experience on Slack, such as granular control over communication settings and tools for admins to designate legal or HR contacts.
  • Policy: Risa's team is responsible for developing and enforcing policies that govern the use of Slack's products. This includes creating guidelines around acceptable use, and working with legal and HR teams to ensure that Slack's policies align with the company's values.
  • Operations and enforcement: This involves monitoring Slack's products for any policy violations, and taking action when necessary to address them.

How does Risa navigate challenges when working with product managers who are responsible for trust, safety, or accessibility?

  • Risa emphasizes that building for trust and safety is central to Slack's ethos, so product managers are usually quite willing to work with her.
  • All product managers are working towards the same goal of understanding customer needs and meeting them.
  • If you lose the trust of your customer, it is very hard to get back, and you're not going to be able to meet that higher order need.

Quotes from Risa Stein in this episode

There's always going to be a new challenge or a new threat that you have to figure out how to meet, and you're continually raising the bar on things like safety and privacy and security. And as a result, you're continually learning, which is really fun. And I've been doing product for a little while now but I still feel like I am learning so much every day in my job as a result of this.
It doesn't matter what platform you're on, it doesn't matter what product you're building. If you lose the trust of your customer, it is very hard to get back and you're not gonna be able to meet that higher order need. And trust is something that has to be earned. It's not just given, it's not a default. It's something you have to earn and earn again every single day.
It felt like working in a field like trust and safety would be a way for me to work towards those goals, to use what I had learned in law school things like persuasive communication, advocacy, and apply them towards technology where I had the potential to me, as just one person, impact potentially millions of people's daily experiences, now that we've reached a point where so many of us are living so much of our lives online. So taking that place where so many of us spend so much of our lives and trying to play at least a teeny, tiny role in making it a safer and more equitable place.

The Product Science Podcast Season 5 is brought to you by Productboard

Check out their eBook: 10 Dysfunctions of Product Management

If your product team is struggling, it could be one of the 10 dysfunctions of product management. Learn more about each dysfunction and how a product management system can help you address and avoid them.

Get Your Copy


Holly Hester-Reilly:                                   Hi, and welcome to The Product Science Podcast where we're helping start-up founders and product leaders build high-growth products, teams, and companies through real conversations with people who have tried it and aren't afraid to share lessons learned from their failures along the way. I'm your host, Holly Hester-Reilly, founder and CEO of H2R Product Science.

                                                       This week on the Product Science Podcast, my guest is Risa Stein. Risa is director of product integrity, leading teams' focus on protecting customers' safety, privacy, and security. Risa and her team are responsible for setting and enforcing policy, investigating and stopping bad actors who try to abuse Slack and developing new product features to protect Slack's customers. Prior to joining Slack, she led transparency and safety experience products at LinkedIn and worked in trust and safety and product at Twitter. Risa received her JD from Stanford Law School, an MBA from the Stanford Graduate School of Business, and her honors BA from Brown University. Risa lives in her hometown of San Francisco with her husband and two dogs, Noodle and Pancakes. Welcome, Risa.

Risa Stein:                                            Yeah. Thank you so much for having me, Holly.

Holly Hester-Reilly:                                   Yeah, I'm excited to have you on. It sounds like you spent a little bit of time in my part of the world when you went to Brown because I'm from the Northeast.

Risa Stein:                                            Yep. I had to, like all native Californians, go out, experience winter, freeze, and then move back to California.

Holly Hester-Reilly:                                   So you decided it wasn't for you, the winter?

Risa Stein:                                            It's beautiful. I saw it snow for the first time when I was, I think, 18 years old. I made it six years and then I was like, "Okay. Back to the sunshine."

Holly Hester-Reilly:                                   Yeah, that makes sense. So I always love to hear from people about how they got into products. So what's your story?

Risa Stein:                                            Yeah. I have an interesting one, a little bit of a, I guess you'd say, non-traditional background for product in that I actually didn't intend on becoming a product manager. So I started my career in finance and risk management. Went to graduate school, as you mentioned, at Stanford, got my law degree, did an MBA. And I remember while I was in business school there, a lot of my classmates who were really interested in this career called product management, that I had no idea what it was. And the way they describe it in business school is, "Oh, you're kind of like a mini CEO of your product." And that was something that by that description really did not appeal to me. So I know that what I'm passionate about is making the internet a safer, more accessible and more equitable place, and it didn't sound like a natural fit for that goal.

Holly Hester-Reilly:                                   But if you don't mind, I have to ask, what made you so interested in making the internet safer and more equitable? Because I feel like that's not such a common passion.

Risa Stein:                                            As I mentioned, I went to law school, but I didn't actually want to practice law. It's, in a lot of ways, a very solitary profession. You're in an office reading and writing alone for many hours a day a lot of the time, especially if you go to a big firm, which is the common path. But I knew that I wanted to still work towards justice and that fairness and equity were really important to me and personal motivators that I wanted to be throughout my career.

                                                       And it felt like working in a field like trust and safety would be a way for me to work towards those goals, to use what I had learned in law school, things like persuasive communication, advocacy, and apply them towards a field, technology, where I had the potential to, me, as just one person, impact potentially millions of people's daily experiences now that we've reached a point where so many of us are living so much of our lives online, whether it's in a personal context, connecting with friends and family, or like with Slack, whether it's doing our job in a digital HQ where we're collaborating and getting work done online. So taking that place where so many of us spend so much of our lives and trying to play at least a teeny, tiny role in making it a safer and more equitable place.

Holly Hester-Reilly:                                   Awesome. I think that's fantastic. I'm personally also very motivated by creating products with product management that impact the world positively. I think that's really great. So you were saying that you wanted to move in that direction, and how did you break in?

Risa Stein:                                            Yeah. So I had this idea of what product was, and I thought it was just like management consulting, which was another job that at that point in time I didn't understand very well. I ended up finding a role that was at the intersection of product and policy and social media and going into it not realizing how deeply tied in with product it was, and once I got there, seeing what it really is that product managers do. It's not about being a mini CEO. It's not about just issuing edicts from on high. It's about being that interstitial glue and just whatever it is that needs to get done, filling the gap, getting it done. So it's about understanding what it is your customers need and doing whatever it is to best serve those needs and connecting with all different kinds of folks across your company in a very interdisciplinary way. So I was very fortunate to trip and fall into this field that I didn't understand at the time but I now realize is an incredible place to be as someone who has very interdisciplinary interests.

Holly Hester-Reilly:                                   Yeah. And at this point in time, you're director of product for integrity at Slack. Tell us more about what does that entail?

Risa Stein:                                            Yeah. So I guess starting just from a little mini overview of Slack, in case your listeners haven't had the chance to use it before. Slack is a digital HQ. Our mission is to make your work life simpler, more pleasant, more productive. So it's really a space to collaborate whether you're in an office or distributed around the world. The integrity team has three broad areas we're responsible for. The first is, of course, integrity product. So that entails building new product features that are specifically designed to protect our customers' privacy, their security and their safety, as well as to safeguard the platform from threats or bad actors. And that's traditional product development, engineering. We also work very closely with product managers across the company to help them engage in integrity by design, so ensure that when they're releasing a new feature for the first time, it's safe and it's protected from potential misuse or abuse.

Holly Hester-Reilly:                                   Yeah. So I guess I'm curious, if we could get into the nitty-gritty of that a little bit, what is a use case, or a product, or a feature? What's a problem that you solved that you can tell us a story about?

Risa Stein:                                            Taking a slight detour here, one thing I love about working on B2B software having come to Slack from a social media background is there's this added kind of fun complexity of two customers. There's your end users who are in the platform all day, every day, collaborating, getting stuff done, but there are also the admins who are responsible for managing their employees' experience and their Slack instance. And so it gives you two different customers to consider and to build for and to bring along in this journey. So that's another thing we've been working on is new settings that make it easier for us to reach out to our admins should something go wrong. And we're in the process of rolling this out, but they can designate a legal or HR contact within their preferences for us to connect with should we need to.

Holly Hester-Reilly:                                   Okay. And going back a little bit, you had started by telling me that there were three areas that you oversee, and we dove into one of them. What are the other ones?

Risa Stein:                                            Yeah. So the other two are policy, so these are rules of the road, what you can and can't do on Slack, and then operations and enforcement. So policy would be around, "You can't share illegal content or engage in illegal behavior on the services. Here's what that looks like." And enforcement would be the identification of violations of those policies and taking action on that.

Holly Hester-Reilly:                                   Okay. One of the things that I think a lot of product managers experience is some frustration or challenges when they're working with someone else in their company who is responsible for trust, or safety, or accessibility, or some of these things that are like table stakes. We just have to do it in order to be a good company or to keep ourselves safe from risk. How do you navigate that with the product managers at Slack?

Risa Stein:                                            Yeah. I'm fortunate in that it's pretty central and inherent to Slack's ethos that we want to build ethically. And so I can't say that I've had any dramatic blowup or conflict with my fellow product managers at the company. I certainly in past lives at other social media companies have run into that kind of challenge you're describing. I think at the end of the day it comes back to we're all accomplishing the same goal, which is to build for and serve our user, understand, "What is the customer need?" design for that need, and ultimately meet that need. That's, at least in my mind, one of the core kernels of product management is you need to understand what it is your user needs and build for it.

                                                       Safety is a huge piece of that. It doesn't matter what platform you're on. It doesn't matter what product you're building. If you lose the trust of your customer, it is very hard to get back, and you're not going to be able to meet that higher order need. And trust is something that has to be earned. It's not just given. It's not a default. It's something you have to earn and earn again every single day. So that's one piece of it. If you don't have trust, if you don't earn trust, if you don't do those kind of table stakes things you referred to, you're not going to be able to do whatever that next goal is, to grow, to expand, to ship new features. So that's the first layer.

                                                       And then the second layer is, again, connecting back to user need. Your most vulnerable users or your most vulnerable customers are still your customers. So if you design a feature that can both meet the need of the "average" or typical user and also better serve a more vulnerable customer, you've only expanded your potential reach. And actually, let me rephrase that. Your most vulnerable users, at the end of the day, are still just your users. They're just your customers. And if you can build a product or feature that serves that most vulnerable user or most vulnerable customer, it will also serve everyone else and serve them better.

                                                       So to give you a good example of this, one thing I love about Slack is it's so friendly to asynchronous work and asynchronous collaboration. And for your "typical user," that might be great because leave have a message for my colleague and explain something that I was working on, and I can just do that super quick in a clip and shoot that off before I run off to daycare to do pickup. And rather than scheduling an hour long call, you still get the information you need, there's still that connection, but I have the flexibility, and something like 70 plus percent of workers want that kind of flexibility in how and where they work. I have the flexibility to get things done on my own schedule.

                                                       But another really cool thing about building for asynchronous communication is it's accessible to folks who are neurodivergent and who for maybe live video communication and a very set, structured video meeting isn't the way they work best, or for whom having a constant stream of notifications or even working in an office environment where they're surrounded by people talking on the phone can be very distracting and maybe those distractions take longer to recover from. So by building something like clips, which allows very flexible asynchronous communication, you're serving the use case for 70 plus percent of users who want flexibility in how and when they're working, that "typical user," but you're also building a feature that can be very empowering to someone who is neurodivergent and maybe needs to work in a slightly different way or have more granularity and the control over the way they work, when they work. You're serving that more... In this case, vulnerable is not quite the right word, but you're serving maybe that more niche user and making a product that's better for everyone and in turn is making this piece of the internet more equitable and more accessible.

Holly Hester-Reilly:                                   Yeah, that's really cool. So one of the things that makes me think about is just the challenges of asynchronous and synchronous communication, and I'm curious if that is impacted by your integrity team's work. How does it overlap with the support of asynchronous communication?

Risa Stein:                                            What is the intersection between asynchronous communication and integrity? Is that the question?

Holly Hester-Reilly:                                   Yes.

Risa Stein:                                            Okay, cool. So integrity is going to look very different everywhere depending on the product and the customer need or use case. So I think we have a common conception of what trust and safety looks like that's very shaped by social media, which is focused on more things like misinformation, hate speech, abuse, terrorist recruiting, all of that kind of stuff. And as you might imagine, those are not the primary issues one would face when working on enterprise software. Instead, my team spends a lot of time thinking about issues like privacy and security but also psychological safety.

                                                       So in order to get the best work of your life done, you need to feel safe at work, and there are ways in which we can't control for that. Slack doesn't know what your relationship with your colleagues is like. We don't know what your relationship with your boss is like, and that's not what we do. But what we can do for our users is empower them with more granular control over the way they work, over how they access communication, over the way they communicate and collaborate back. And that control and flexibility, that empowerment really helps contribute to having more control is empowering, and it helps create greater psychological safety in collaboration. Does that make sense?

Holly Hester-Reilly:                                   Yeah. I think I understand what you're saying. So it's about being able to control what some of the access or settings look like and being able to feel more safe in communication because you know that there is that level of control.

Risa Stein:                                            Exactly. It's not only about settings, and I think settings are a big piece of this. It's really helpful to be able to, for example... I need to do some focus work, or maybe there's something I need to just take a step back from for a second. I can, "Do not disturb," and shut off all my notifications, or I can mute a channel that is particularly disruptive to me. Those settings are powerful, but it's also in the little things like with huddles. I love huddles and I love the hold music, but not getting distracted by that. When you join a huddle, video's off by default, and the focus is instead on that thread where you're taking notes that are recorded and kept after the meeting so you don't lose them.

                                                       But it's in the little things like without video on by default, there's less of that focus on, "Oh, how's my makeup? How's my hair look. It's still wet. Oh, no, my dog is in the background being embarrassing," whatever it might be. The focus is more on the conversation I need to have with my colleague real quick, "Let's figure the X, Y, Z out and get this done and get it out the door." That kind of little detail does contribute meaningfully towards psychological safety I think at work. Yeah, you can turn your camera on if you want to, but you can also just have a really quick touch base and move on with your day and have a little less pressure. So that's the kind of thing that maybe in a social media context you would never think of as being related to integrity, but in a work context and your digital IQ, that's an example of a kind of flexibility or personalization in your work environment that is very empowering.

Holly Hester-Reilly:                                   So you've mentioned a couple of times how things are different being in a work environment. So take us back a little bit. How were things different working in social media?

Risa Stein:                                            So it is very different, and also just from a broader standpoint, one of the things I love about product management is that this job looks so different depending on where you do it, what you're working on and who you're doing it with, which at least I find keeps it really exciting because the core skills required to be a PM in social media and to be a PM on B2B enterprise software are similar, of course, but it's like a whole fresh new thing and trying something totally different. So there's always opportunity to learn. Working in social media, there's some things that are generally very different, and then there's some things that are different specific to trust and safety.

                                                       So from a general standpoint, when I was working on social, you're just constantly putting out lots and lots of little A/B experiments, testing things out because you have millions or hundreds of millions of people pretty much instantaneously interacting with your product, you have immediate feedback, the experiment's red or green, and you're just constantly iterating. That looks a little different in a B2B context, right? And it gets back to something we were talking about earlier, which is that it's about more than just you and the end user. It's about you and two kinds of end user, two kinds of customer, the person who's getting their work done in there and the admin who has to manage this workplace, manage this digital HQ. So they need to understand what changes you're working on. For example, think about how disruptive it would be if you go to work in the morning and you log in to Slack and overnight everything has changed, everything has moved. You have no idea what's where and what's going on. It would be super disruptive.

Holly Hester-Reilly:                                   Yeah.

Risa Stein:                                            That would not be good.

Holly Hester-Reilly:                                   No one would like that.

Risa Stein:                                            It would make it really hard to get your job done, and we don't want that. We want to make it simpler, more productive to get your work done. So communication with the customer in making changes I think plays a much bigger role. The way you interact, you have the opportunity to interact much more directly with the customer. And of course, in both contexts you're doing user experience research.

Holly Hester-Reilly:                                   Tell me more about that direct relationship that you have with customers at Slack. What does it look like?

Risa Stein:                                            At Slack, we're talking to our customers constantly, and we have this really direct relationship, which is very cool. Rather than mediated through a series of A/B tests where sometimes the user is a number on a screen, we'll have direct conversations when we're thinking about a problem space or an area and reach out to an admin for, say, a large company that's a customer and have the opportunity to sit down with them and understand directly the pain points that they're facing as someone who's responsible for managing this community, but also that their users, their employees, who are getting work done in the workspace every day, what their experiences are.

                                                       And in at least my own experience with social media context, that happens much more through the context of a long-term user experience research mediated study where you're reaching out and offering incentives, and it's more distant. And you might interview that person and then never speak to them again. Whereas in a B2B context, you have a relationship that could go on for many years with the same customer, and they can see things change over time and see the impact of their feedback on the product.

Holly Hester-Reilly:                                   Yeah. That's one of the things I love about working in B2B as well. I'm curious to hear what specifically does that look like for Slack? Do you send Slack message to the customers? Do you hop on a huddle? Do you get on a video call with another or without Slack?

Risa Stein:                                            Maybe, unsurprisingly, we are Slack super users over here. I do 95% of my workday in Slack, which is pretty awesome. And talking to customers is no different. Slack Connect is the way you can have shared channels or conversations with folks who work at different companies, share communication within your workspace. I'm sure that the Connect team would not be pleased with that description. They would give you a much better description. But functionally, Slack Connect is the way you can connect with folks who work at other companies, and we use Connect to reach out directly with our customers. When we have those questions, we'll just shoot off a quick DM to the customer and to our internal team member who works with that customer, and we can have that conversation in real time in Slack. And then Slack also has customer advisory board with lots of customers who participate in regularly giving feedback in a more structured way. But there is that very direct back and forth communication directly through the product about the product and how we can improve the product.

Holly Hester-Reilly:                                   So what does that play when you decide whether to go through the customer advisory board or do a direct communication with the customer that you've got a relationship with?

Risa Stein:                                            Yeah. So part of it will depend on the question we're looking to answer. So sometimes we'll have inbound feedback that a particular customer is facing a certain type of challenge, and that's where we'll want to have a conversation one-on-one and understand the need and understand how we might be able to better build for it. Or it could be we're working on a certain type of feature that is most relevant to, say, a very large organization with tens of thousands of employees. We'll go to certain customers where we know that they're running that kind of workspace, they have that kind of global presence.

                                                       Or if we have a more broad topic where we're just looking for feedback from all different types of customers, let's say we want to talk to folks who are in all different types of industries, who work on healthcare and finance and consumer goods, whatever it might be, that's where we're going to reach out to that broader group of customers and get maybe all different types of insights that you don't know what you're going to get back, but it's fun, and you'll see how different features come into play differently depending on what it is they're building in their workspace, what their product is.

Holly Hester-Reilly:                                   Yeah. I don't think I've had a lot of guests on that work at B2B companies that have the scale that Slack has. And so, one of the things that I'm curious about is what does the operations of product and UX research look like there?

Risa Stein:                                            Yeah. The scale at Slack is crazy.

Holly Hester-Reilly:                                   Yeah.

Risa Stein:                                            That's one thing people tell you going from social media to B2B, there's this idea that you're no longer solving it at scale. But Slack at peak times, we're handling 300,000 messages a second. And trillions of messages have been sent on Slack, which is definitely at scale. And particularly from a trust and safety lens. If there's a one in a million chance of something, that sounds pretty rare, but a one in a million chance happens decently often when you're talking about 300,000 messages per second. Every couple of seconds you have a one in a million chance.

Holly Hester-Reilly:                                   Absolutely.

Risa Stein:                                            One thing that Slack does really beautifully is our customer experience team, which is how we handle operations and all of our communications with those customers. And actually, that team customer experience is one of our closest partners on integrity because they're the ones who are caring about it first, whatever it might be. Whenever something goes wrong, that's how it's going to come into us. And I think the reason Slack's, in my personal opinion, been able to handle that kind of scale of interaction with the customer so is because the customer experience team is really focused, at the end of the day, on just human communication.

                                                       If you look out there, this is one of those things I looked at when I was considering joining Slack, there's this great piece on Slack's voice. One thing that's been really important to the customer experience team and as they've scaled out those operations is just we are human beings talking to other human beings who are just trying to use this product. And don't lose sight of that. Not everything has to be super polished and perfect and sound like it's right off a script. At the end of the day, we're all just people talking to other people, and let's do whatever we can to help them solve whatever problem it is, or bug they're running into, or challenge they're having, getting logged in, and just handle it like people.

Holly Hester-Reilly:                                   Yeah. So the document or writing that you're talking about was something about the voice of Slack, how people write for Slack?

Risa Stein:                                            If you ever have the opportunity to react to a Slack customer experience and reach out to feedback@slack.com, it is really amazing just how much you don't feel like you're talking to a customer service robot. Because you're not. You're talking to someone who works at Slack and wants to help. And I think that also, even though it's not part of integrity, it also gets towards that broader ethos Slack has around really genuine just human connection. And feeling like you can reach a real person when something goes wrong I also think goes towards a sense of psychological safety in engaging in this digital workplace.

Holly Hester-Reilly:                                   Yeah, absolutely. Maybe we could switch gears a little bit. I'd love to hear a bit more about your time in social media, and I know that you worked at LinkedIn for a while.

Risa Stein:                                            Yeah.

Holly Hester-Reilly:                                   What was your role there, and what was it like?

Risa Stein:                                            Yeah. So when I was at LinkedIn, I worked on all things content moderation. So that included redesign of the way users would report content that they believed to violate LinkedIn's policies through to how LinkedIn communicated back with those reporters after content was reviewed and moderated, as well as the other side of that experience, which is if you ran afoul of LinkedIn's rules and posted something harmful, for example, that was taken down, what was your experience like? Could you reach back out and appeal and get a second look? That whole process as well as content moderation related regulations globally and compliance with those.

Holly Hester-Reilly:                                   So what is content moderation like at LinkedIn? I'm a LinkedIn user, I'm on it almost every day, but I don't really know. How does that work?

Risa Stein:                                            Yeah. It's a similar story. When you get to that kind of scale, eventually people are just people and sometimes people make bad decisions or act maliciously. Any social media platform, this is going to be true. There will be issues like misinformation, there will be hate speech, there will be abuse, all of that kind of thing. So generally with social media, the way content moderation works is there's some user mechanism to flag content to the platform as well as proactive detection, ML models that will screen content and identify something that looks like it violates the policy. There are human moderators who will review that content and decide whether or not it violates the rules and then take action on that content, which could mean taking it down, taking an entire account down if someone has severely violated the rules and communicating that back to the user.

Holly Hester-Reilly:                                   Yeah. So when you were working there, did you face any challenges that were significantly different from what we've talked about at Slack?

Risa Stein:                                            I think working in trust product, there's a lot of commonalities no matter where you do it, but certain issues are thrown in sharper relief depending on the circumstances of the platform and the product. So one example of this is the role metrics play and the type of metrics you're looking at. In a social media context, you're very responsive to user metrics and you might care about something like, "How many users are spending how much time on my platform?" At Slack, of course, they're like, "We are very metrics-driven as a company," but the metrics look a little different. We care about, can you get work done in a simple way? We don't want you to spend time working just so you spend time working. We want to help you collaborate and get your work done. But that emphasis in social media on metrics engagement can be tricky in a trust context because sometimes harmful content can be very engaging. But also, a lot of the metrics that immediately pop to mind when you're thinking about content moderation are attacker-driven metrics.

                                                       So earlier we talked about how one thing that's really interesting working at B2B software is there's three players on the board. There's you, there's the end user, there's the admin or the person managing whatever the product is. In fact, there's actually four players on the board in B2B, and there's three in social media. And that third player for social media and the fourth player for enterprise software is the bad actor. There's this adversarial relationship, and that bad actor plays a role in moving those metrics, which makes it hard sometimes to set actionable goals or targets.

                                                       So to give a specific example, let's say I was working in social media, let's say I was back at LinkedIn and we are redesigning the reporting flow, which is one thing I worked on while I was there. So this is, you see something that maybe it's harmful content comes up in your feed. You click those little three dots in the top right-hand corner and you get to a button that says report. You click on it and it asks you, "Is this adult content? Is this hate speech? Is this misinformation?" whatever it might be. "Is it spam or scams?"

                                                       You might think, "Oh, one good measure of how well the report flow is working is how many reports we're getting." So it might be," I wonder if we make the report flow better will we get more reports?" And maybe you would. Maybe you'll see drop off lower because it's easier to understand. But how many reports you get isn't just a function of how discoverable your report flow is or how motivated your target customer, that end user is to report. It's also a function of the prevalence of harmful content on your platform, and that's not determined by you necessarily. It's not determined by that end user you're trying to serve. It's this potentially bad actor who is, say, flooding your platform with lots and lots of spam.

                                                       So at any given point in time, you could see far more reports not as a function of what a great job you've done redesigning that report flow but as a function of there's a massive spam campaign targeting your platform. So that's an example of an attacker mediated metric where it's more complicated than, say, if I were an onboarding PM and the metric I was looking at was successful completion of the onboarding flow, where there's much more of a direct relationship between changes I make and the ultimate metric at the end of the day with something like reports, or blocks, or lots of things you think about related to trust and safety. There's an attacker element to the metric, which makes it hard to set predictable targets over time.

Holly Hester-Reilly:                                   Yeah. That's really fascinating. It sounds like you've seen changes in just attack volume. What is that caused by?

Risa Stein:                                            It could be all different types of things. In social media, trust and safety, what you're working on ends up being very responsive to what's happening in the world. So for example, I got shaken awake earlier this week by an earthquake. That's the downside of living in California. Let's say hypothetically there's a big earthquake and there's power outages and there's damage, and that's the kind of environment, a natural disaster, in which you could see a spike in misinformation. Or another example, let's say I roll out a new feature that relates to payments, makes it really easy to accept or send payment over my hypothetical social media platform. That's a new attack surface if I'm a scammer. So you might see a big spike in scammers because all of a sudden there's this new means of getting paid or taking advantage of unworthy customers. So it can be for many different reasons. It might also just be that someone hadn't thought of trying to attack you in a particular way before, and they made their way down the list and got to you.

Holly Hester-Reilly:                                   Right.

Risa Stein:                                            One thing that makes trust product in particular so exciting is that there's this adversarial aspect to it where you never totally solve something. There's always going to be a new challenge or a new threat that you have to figure out how to meet, and you're continually raising the bar on things like safety and privacy and security. And as a result, you're continually learning, which is why it's really fun. And I've been doing product for a little while now, but I still feel like I am learning so much every day in my job as a result of this.

Holly Hester-Reilly:                                   Yeah, I love that. I think maybe that's a good message to end on because we're all about learning and growing and changing every day. I think we're about out of time, so I would love to wrap this up with how can people find you if they want to learn more about Risa?

Risa Stein:                                            Yeah. You can find me on LinkedIn, Risa Stein. I have a weird name, so that's useful. There are not a ton of Risas out there. But yes, please find me on LinkedIn. Happy to connect.

Holly Hester-Reilly:                                   Awesome. All right. Thank you so much. It was lovely to have this conversation, Risa.

Risa Stein:                                            Thank you.

Holly Hester-Reilly:                                   The Product Science Podcast is brought to you by H2R Product Science. We teach start-up founders and product leaders how to use the product science method to discover the strongest product opportunities and lay the foundations for high-growth products, teams, and businesses. Learn more at h2rproductscience.com. Enjoying this episode? Don't forget to subscribe so you don't miss next week's episode. I also encourage you to visit us at productsciencepodcast.com to sign up for more information and resources from me and our guests. If you like the show, a rating and review would be greatly appreciated. Thank you.

More Posts