Machine learning and the incredible flying robot with Dr. Ashish Kapoor

Autonomous flying agents – or flying robots – may seem like the stuff of sci-fi to the average person, but to Dr. Ashish Kapoor, Principal Researcher and Research Manager of the Aerial Informatics and Robotics Group at Microsoft Research, they’re much closer to science than to fiction. And, having built – and flight tested – his own airplane, complete with state-of-the-art avionics designed to run AI and ML algorithms, he has the street cred – or should we say flight cred – to prove it.

Today, Dr. Kapoor talks about how cutting-edge machine learning techniques are empowering a new generation of autonomous vehicles, and tells us all about AirSim, an innovative platform that’s helping bridge the simulator-to-reality gap, paving the way for safer, more robust real-world AI systems of all kinds.

Ashish Kapoor: So the way I see my airplane, the technologies are already there. If I could just solve landing and take-off, that would be a flying car that I would bet my life on.

Host: You’re listening to the Microsoft Research Podcast, a show that brings you closer to the cutting-edge of technology research and the scientists behind it. I’m your host, Gretchen Huizinga.

Host: Autonomous flying agents – or flying robots – may seem like the stuff of sci-fi to the average person, but to Dr. Ashish Kapoor, Principal Researcher and Research Manager of the Aerial Informatics and Robotics Group at Microsoft Research, they’re much closer to science than to fiction. And, having built – and flight tested – his own airplane, complete with state-of-the-art avionics designed to run AI and ML algorithms, he has the street cred – or should we say flight cred – to prove it.

Today, Dr. Kapoor talks about how cutting-edge machine learning techniques are empowering a new generation of autonomous vehicles, and tells us all about AirSim, an innovative platform that’s helping bridge the simulator-to-reality gap, paving the way for safer, more robust real-world AI systems of all kinds. That and much more on this episode of the Microsoft Research Podcast.

Host: Ashish Kapoor, welcome to the podcast.

Ashish Kapoor: Yeah, thank you. I am excited to be here.

Host: Great to have you. You’re a Principle Researcher and the Research Manager of the Aerial Informatics and Robotics Group at MSR. Give us a picture, in broad-brush strokes, generally, of what your group does.

Ashish Kapoor: Our group focuses on robotic agents, and more specifically flying robotic agents, that can positively influence the society. So, we are focusing on how the current advances in robotics, machine learning and AI can help us enable that.

Host: So, talk about this phrase, Aerial Informatics and Robotics also known by the acronym AIR, which is delightful. It’s a group and is also a platform, yes?

Host: Tell us about that.

Ashish Kapoor: So, when we started a few years ago trying to build these agents, this is an incredibly difficult area to work on. And one of the reasons is the amount of effort that you need to make sure that the robotics systems work well in the environment is huge. You are not just dealing with software. You are dealing with hardware. You are dealing hardware and software integration. You are dealing with a field study that you need to make sure that all your robotic components, all your software components, all your hardware components, all the sensor components work well. You need to make sure that, in the presence of external environments, such as exogenous variables, such as rain, wind, you know, maybe other human users nearby, animals, birds… So, the amount of progress was very tiny compared to the amount of time that we were spending on it. And we quickly realized that what we need to is, do we need to come up with a simulation platform that would let us recreate these elements in the lab setting so that our end-to-end cycle between development and testing gets reduced rapidly. So consequently, we build this platform, uh, simulation platform, and we released it under the name of Aerial Informatics and Robotics Platform.

Host: So that’s the platform there.

Ashish Kapoor: And what it is, it’s a simulation for AI to build autonomous agents and train machine learning systems. It’s an AI simulation, that’s how I think about it. So, group is Aerial Informatics and Robotics platform. And it is really about flying robots, you know, and they are powered by machine learning. They are powered by AI techniques. They are powered by intelligent human-in-the-loop methods. So, we constantly think about how easy would it be for a pilot to actually fly these drones and carry out missions which might be risky, which might be ambitious?

Ashish Kapoor: But at the core of this group lies this philosophy behind simulation. We think that simulation is one of the foundational pillars that would enable us to build these kinds of technologies.

Ashish Kapoor: A simulation like this allows us to rapidly build, prototype and test out our technologies. But it also allows us to make sure that we make the mistakes in the safety of our labs. It allows us to make sure that our softwares – the bugs have been ironed out before we go and fly the real thing out in the world.

Host: When I think about flying robots, it usually has somebody behind the controls, right? And so, you talk about humans-in-the-loop and that they are always going to be there, but yet it seems like you’re moving towards a more and more autonomous flying robot…

Host: How is that different from autopilot today?

Ashish Kapoor: I would claim that, you know, there are already been flying robots for a while. And your commercial airliners… it’s a giant flying robot! In fact, most of the flight is actually carried by an autopilot these days. And we are comfortable with the technology and of course, there’s human-in-the-loop, there are pilots manning the cockpit, and they are making sure that things are correctly set up for instance. So, what has actually changed is, in the recent past, all these technologies are now available for drones as well. So, it’s exactly the same technology. It’s pretty much path following. But when we think in the context of drones and gliders and other objects that everyday people can just go and fly, you know, the stakes becomes much higher because now you are closer to ground. There are other people in the vicinity. There are buildings. There are trees. And, the potential for anyone to just pick it up off the shelf and go and fly? I mean the risks are huge.

Ashish Kapoor: So consequently, we need to start thinking a little bit more about safety, about how these AI technologies can positively influence our world, but also be safe while doing so.

Host: Tell us about the flight of your AI-controlled autonomous infinite soaring machine that you did last summer. I watched the video. It was really cool. What was that about, and why is it important that we have these autonomous infinite soaring machines, gliders.

Ashish Kapoor: That project… this was more of an AI aspirational project. And if you think about some of the recent breakthroughs in AI, they have things like reinforcement learning, deep reinforcement learning, or deep machine learning algorithms. But much of this uh, breakthrough has happened in the software space. When an agent needs to make a next move, they don’t have to worry about exogenous variables, etc. And the second thing is because the inherent software nature of these problems, it’s possible to potentially generate near infinite amount of data. And that’s what many of these techniques are banking on. So, we started thinking about what would it take us to actually think about similar problems but in the domain of real-world AI? And this autonomous glider is basically, if you think about it, it’s kind of a, you know, a game, but you are playing this game in a real world. So whole goal in the autonomous glider project was to design an autonomous flying agent that can travel large distances, or that can stay in the air for long amount of time. And the way it does it, it has some information about the terrain, some information about the weather. It has some statistical model of how the rising air bubble work. And it exploits all this information in order to make a decision whether it should hang out at a place and wait for the raising air to carry it up and gain altitude, or it should go to another place. And while in doing so it needs to tradeoff between how high it’s flying and how far, and what’s the probability of finding such a bubble of rising air. And by doing such kind of tradeoffs, it’s possible to keep the glider aloft for a long amount of times. However, this is now a real-world problem. We don’t have the luxury of obtaining near infinite data. And in addition, we have to deal with exogenous variables, possibly variables that we have never counted for in our algorithms. So, if we could solve this problem, this will definitely be a big step in the development of AI and ML techniques.

Host: So, the glider doesn’t have an engine. You send it off. And then what it’s doing and learning is how to stay aloft for a long period of time without any external force other than nature.

Ashish Kapoor: The nature, exactly. So, the glider doesn’t have any external propulsion system. All it relies on is these bubbles of rising air that it hopes to piggyback upon to get an altitude. And then really the decision that the glider has to make is where to go in order to obtain the maximum lift but also such that it doesn’t veer too far away from its core mission.

Ashish Kapoor: So, it’s a tradeoff between exploration and exploitation that we talk about in machine learning.

Host: Well, let’s talk about the role of simulation in AI for a bit. You refer to this as “high fidelity simulation to enable real world AI.” And you’ve even said it represents the next generation of machine learning models. So, how and why is simulation important to your work and to the larger field of AI?

Ashish Kapoor: You know, as you mentioned high fidelity simulation, I think, is going to be one of the most critical component of delivering real-world AI systems. Let me tell you the top two reasons, right. So, the first reason is that, if you look at the recent advances in machine learning and AI, these advances are very data-heavy. We have spent billions of dollars in thinking about computer architecture. We have looked into advances in algorithms in order to make machine-learning systems fast. But a critical component to any of the efforts is the existence of data. If there is no data, none of this is going to matter. Right now, data is really the bottleneck. You have many startups, you know, many labs trying to just annotate and label the data for you. So that you can build richer and richer models. So, the scarcity of the data is one of the core reasons that simulations are going to become important any more, so the first reason. The second reason is that, if you look at you know recent advances, is reinforcement learning algorithms. So, any of the games, any of the arcade games, those are perfect simulation to the situations your agent will be playing in, and they bank upon it to get more and more data, to gather more and more experiences, and build richer and richer and better models. However, in order to utilize similar techniques for building real-world AI systems, we would need similar kind of simulation and there doesn’t exist a simulator that would make the real world.

Host: Well, let’s talk about that creating reality in software, which is basically what simulation is, right?

Host: You’re using software to bring a real world into play. And you said that machine learning simulation models are like video games on steroids.

Host: What do you mean by that, and what role does reinforcement learning play in that?

Ashish Kapoor: Yes. So, you know, when I think about simulation, especially simulation to train AI and ML agents, I think about… it’s a video game. And as I said, it’s a video game on steroids. And uh, video game on steroids that’s played not by a person, but by a software, a machine learning and AI agent that expects to explore and learn from those experiences, and build better machine learning models. In fact, if you look at the recent work in reinforcement learning, in deep reinforcement learning for instance, they are actually playing games. They are playing Atari games. They are playing Pac-Man. And they are going to play it on and on for millions of times and get a lot of data and get better at it. So, imagine if we could create a video game, which is, you know, mimicking reality in many different aspects that we wish to deploy our AI and ML systems in, and we let these software agents play on and on, and gather valuable experience, see all the corner cases, we can expect that these agents will start to get better. Build better machine learning models which then can be deployed in real world. This is not your average video game. The simulation is a game where we are not only simulating the virtual world and the character, but we also simulating each and every sensor that your autonomous agent is going to see. All of them are being simulated to a detail, such that the kind of data that your video game is providing is very similar to the real world.

Ashish Kapoor: So consequently, if you’re training a system end-to-end in simulation, you can expect that a lot of that knowledge will be transferred to the real world. So, yeah, reinforcement learning is a very interesting technology. And you know if you think about it, how would you take reinforcement-learning algorithm and maybe train a car or a drone to fly with it? You know, naïve application to that technology in real world is going to be disastrous. Why? Because if your car makes an error, very likely, it’s going to be a costly mistake. Just to give you a statistic, so for instance, if you look at any of these, you know, reinforcement-learning games that have been played and that have been successful, the amount of data that has been pushed through the system is incredible. So, for instance, for you know, something like Pac-Man, you are looking at hundreds of millions of frames before your agent starts to do something that’s reasonable. And you know, like hundred million frames, at 30 frames per second, is more than a month of compute that’s needed. That’s going to be expensive for the real-world tasks, so consequently, if you had an access to simulation where these mistakes didn’t cost you anything except for the compute cycle, maybe, your learning would be very efficient. So, all of these advances in deep reinforcement learning would be, you know, very tangible for the real-world applications, if there exist a simulator that could mimic the real-world behaviors of the vehicles, as well as the consequence of all the actions those vehicles would take, you know, when they are out there.

Host: Tell me a little bit more about AirSim and what it’s doing to bridge the simulation-to-reality gap, and why do researchers need to stress test their algorithms?

Ashish Kapoor: So, as I mentioned, right, the AirSim was really a bi-product of our efforts in building real-world robotic systems. And um, you know, we quickly realized that in order to be efficient we need to come up with a simulation platform that mimics reality as closely as possible. So, we embarked on a journey to build a simulation platform that’s near-realistic but also caters to the problems in the domain of AI. But the thing is, there exists no other simulator that can mimic behaviors of AI systems. And that is what AirSim aspires to be. It says that if your goal is to train a real-world AI system, let’s build components, so things like actuators, the worlds, the trees, the pedestrians, other exogenous variables like wind. All the sensors you might employ… magnometer, barometer, accelometer, gyroscopes, cameras. Let’s simulate everything in detail as much as you can, and then you can simulate the entire AI system, end-to-end, from collection of training data to actual deployment, validation, and testing. We will try to make the simulation as close to reality as possible. So that, when you deploy in the real world, a lot of the learnings in the simulation world gets transferred. This is what AirSim is. It aspires to be a simulator for solving hard AI and ML tasks for real world.

Host: That’s so important for multiple reasons because when I hear the word, AirSim I think of like an airline simulator, right? Just you get in a cockpit; you’re flying. That’s not what it is about.

Ashish Kapoor: You know, it started as a simulator for aerial robots. This is an open source project. And as soon as we open sourced, we started to get a lot of requests to extend it to other domains such as,, you know automobiles and uh, you know other scenarios such as farming, warehouses, etc.

Ashish Kapoor: So, given that, the foundation of AirSim was really about thinking about AI and machine learning system. It didn’t take much effort for us to extend it and open it to other possible AI applications.

Ashish Kapoor: So right now, AirSim is not just limited to aerial robots. In fact, it has a very rich model of ground automobiles. It has ability for you to create vehicles and robots of your choice, you know, we have constructs, we have extensibility, built into AirSim so that you can design your own robots, equip them with the sensors that you prefer, so things like multiple cameras, multiple gyroscopes. You can place them on the chassis as you wish, and then let them to go in the AirSim. You can even create different worlds. With AirSim we also released a bunch of pre-compiled environment binaries which include cities, neighborhoods, suburbans, African Savanna, mountainous environments, and many other such examples where you can deploy your robot and see how they interact.

Host: How about snow on the freeway?

Ashish Kapoor: Yes, we can do that as well. We have at least one environment where, you know, your car, you can basically drive off-road, and it’s mountainous, and has all those problems with unpaved roads, etc., and that’s one of the advantages of simulation. So, for instance, what we can do is we can not only simulate cities and behaviors of your vehicle, we can actually simulate time of day. We can simulate whether it’s raining, how much is it raining, whether it’s snowing. All of this can be simulated. And that leads to much more richer experience for a vehicle, in a simulation, than possibly in the real world. So, you know, at the lower level, these problems have been looked for a long, long time, you know, your brake assist, your park assist. These are the things that are already in a pretty, you know, mature stage of production. However, what’s lacking is the decision-making ability. So, things like, you know, how do I expect the other car to drive? If I’m at a stop sign, when should I go? If I see a pedestrian at a certain distance, how should I act? So, all of these high-level decision-making tasks are the hard ones that humans right now are doing, and that’s where most of the car technology lacks. So what AirSim aspires to be? It aspires to be a platform where you can try, and train, and test and validate, those high-level decision-making tasks.

Host: So, let’s talk about safety in AI systems. What are you doing to tackle this uncertainty problem in the open world?

Ashish Kapoor: You know, as I mentioned earlier, any of our robots, autonomous cars, or flying systems, or any IoT device that’s AI-enabled, will have certain kind of a perception system and it’s never going to be foolproof. There will always be exogenous variables that we have not modeled. So, there will be cases in which it’s going to make a mistake. So, consequently, the challenge is, how do you then come up with strategies, or algorithms, that would work safely despite knowing that their perception system might sometimes fail, right? One of the key insights that we are banking on, is that if we can build a machine learning system that has an understanding of when it’s working correctly versus when it is going to make a mistake, we can then come up with a strategy that’s safe. So, it’s akin to, say, a flying robot that decides to fly fast when it’s confident about its surroundings. But when it can’t see things when it’s raining, when it’s hazy, it decides to slow down in order to think more. So, we are investigating strategies such as that where we embed machine-learning systems that are aware of when they are working well, versus when they are going to make mistakes. And, again, you know, at the core of all this is actually simulation. If you’d like to think about a brain of an AI agent, we simulate all different possibilities that we think are plausible given what the agent has seen in the world. And after running many of these simulations, we then make a choice between taking a decision that might be aggressive, versus something that will keep us safe.

Host: And we see that in our own lives. Humans do that every day.

Ashish Kapoor: Yes. Yes. Yes.

Ashish Kapoor: Yes. And of course, you know, you imagine things on how things will be. If you think about our imagination as a simulation portal where we can reason about how things might be if you took certain actions. And you want our machine learning and AI agents to have access to such imaginative portal as well where they can introspect, where they can analyze the consequences of actions that they might want to take.

Host: Yeah. And that leads me back to this open-source nature of AirSim. Why is open-source so important for this particular platform? And in research in general? I’m seeing it more and more.

Ashish Kapoor: We released AirSim as an open source platform in February 2017. And one of the reasons for making it open source is the fact that what we’re trying to accomplish with AirSim is of AI-complete nature and let me explain what do I mean by AI-complete nature. What we’re trying to do is we are trying to create reality. And, you know, to create reality is to understand reality. So, it’s a seemingly-impossible task that a small group of people cannot accomplish. So we hope that by open sourcing it, we would excite, you know, many researchers, students, developers, you know, who want to participate and contribute back to this project with realistic model of vehicles, sensors, agents, environments. So that we can bootstrap and build something that we individually, ourselves, couldn’t build. And since open sourcing, we have had an active engagement with developers and researchers all around the world, with many of them contributing in very significant ways.

Host: Tell me how collaboration happens in your world?

Ashish Kapoor: As you can image, our work is of very interdisciplinary nature. The team comprises of roboticists, machine learning researchers, developers, engineers, pilots… And the thing is, when you think about such an interdisciplinary project, you need to, you know, realize that many of these collaborations have happened very spontaneously, like over lunch or a coffee or tea. It is clear to all of us that given the ambitious nature of some of these projects, we need to work together, and we need to actively go and seek out folks who are accomplishing some things that can add and complement to our research.

Ashish Kapoor: One of the great thing about this place is that there are hardly any boundaries. And many of our research projects have been a result of an informal hallway conversation or a conversation over lunch or dinner where pretty much we talk about what we doing, and then an idea clicks and we go ahead and build it.

Host: Your background is really interesting. You’re an AI researcher, but you’re also a pilot, and a flight instructor and you’ve actually built an airplane. How did aviation and machine learning research come together in your life?

Ashish Kapoor: I was always enamored by anything that flies, from a very young age, and you know, computer science was, uh, important thing as well growing up. You study computer science because, growing up in India, there weren’t many opportunities for aviation, but you know, computer science was an upcoming field and I was always in love with algorithms and AI specifically. Ended up coming U.S. and did my PhD in machine learning. Once I started working at Microsoft, I realized that it’s time for me to go back and see if there is something I could do in aviation. And, you know, Seattle is the birthplace of aviation… that’s how I think about it. I got my private pilot’s license in 2009, and uh, quickly realized the potential of technology and, specifically, the role of AI and ML, that can possibly change the field of aviation. In fact, you know, beyond autonomous cars, there’s so much discussion on autonomous flight, autonomous personalized vehicles, that can possibly take you from here to say Seattle downtown in less than 15 minutes.

Ashish Kapoor: And to be honest, the technology is maturing very rapidly. And the way I think is that AI and ML are going to one of the critical components to enable that. So, you know, if you think about a self-driving car, a self-flying car is even a harder thing to do, and consequently, we need to make sure that those vehicles are perfectly safe, and they would be pretty much piloted via an AI and ML systems.

Ashish Kapoor: It should be as simple for anyone to go in a car, put an address, and just relax and sit, and that vehicle should go, and drop you there. And in order to enable that scenario, AI and ML would be the core engine that would make this thing happen. This is what I had been thinking about, and through our informatics and robotics research program, we have been trying to answer some of these questions. I think we are at a cusp of technological breakthrough in that space. There’s already, you know, many efforts in the space that are trying to accomplish it. So, you mentioned the airplane that I built, right.

Ashish Kapoor: So, you know, I ended up designing the avionic system as well, and it is designed to run AI and ML algorithms on it. It has sensing for other vehicles in the vicinity. It has onboard weather. It has an autopilot system that can maneuver it not just horizontally, but also vertically. But more importantly, we have the ability to interface that airplane with a planning or a robotic decision-making system that can guide that airplane to fly, you know, autonomously, at least in the cruise phase of flight. So, the way I see my airplane, the technologies are already there, if I could just solve landing and takeoff, that would be a flying car that I would bet my life on.

Host: That’s the first and last mile problem of trucking, or cars.

Host: It’s like it’s just that last bit where the tricky parts come in.

Ashish Kapoor: Yes, exactly, and I think those are the hardest parts. Because now we are very close to the ground, in the vicinity of population and other moving vehicles, and we need to make sure we do a better job in those spaces.

Host: Have you flown the plane you built?

Ashish Kapoor: Yeah. Yeah. Yeah. So, I flight-tested it. I did the first flight and probably it was the scariest thing that I ever did. But it also taught me, you know, you pretty much build your own airplane and then push it through the certification, as well as the flight testing regime. And it gives you a sense of what it takes to build robust systems. I’d like to contrast the state of AI systems to aviation systems say a hundred years back. It’s pretty dangerous to fly, but as a race, we not only enabled that technology, but we came up with protocols. We came up with processes that took that technology and matured the whole technology, and now, you know, we can do things that, possibly a hundred years ago, we never would have imaged.

Ashish Kapoor: And that is what the state of AI is. It might seem very scary. There a lot of unknowns, a lot of uncertainties. But there are a lot of positive benefits of the technology. And uh, I think together we should strive to think about processes and regulations so that we can start using these technologies in beneficial ways.

Host: So that kind of leads into the question I ask every researcher that comes into the podcast booth. Is there anything about your work that keeps you up at night, in terms of, you know, possible downsides or unintended consequences?

Ashish Kapoor: So, when you think about AI and machine learning, machine intelligence technologies, there is a lot of skepticism about, you know, potential negative effects. And a lot of this fear and skepticism, I personally believe, is coming because of lack of proper understanding. So, as we move forward, we need to start debunking some of the myths, and education and knowledge dissemination is one of the ways to do that. And this knowledge dissemination needs to go all the way to the lawmakers, the regulators, so that they can make better decisions. You know, as I mentioned, AI is a pretty powerful technology. You can utilize it to do social good, but you can also use it to do harm. You know, it’s the prospect of doing harm that raises a lot of fears and skepticism in everyone’s mind. Which might further affect how laws and regulations are being made.

Ashish Kapoor: So right now, we are seeing that the advances in AI are happening at a break-neck speed. However, many of these advances, in order to be realized in real world, in order to start affecting our lives, we have to start thinking about regulatory and the law aspects of it. And there is a chance that the regulation and the law-making might lag the amount of innovation that is happening in the AI space. So consequently, I worry about, that possibly, you know, regulations, which might be a knee-jerk reaction to certain situations, might lead to stifling of innovation in our space. And so consequently, thinking about mechanisms in which we can keep innovating as well as think about consequences in society so that we can come up with laws and regulations at a pace that can match the innovation space.

Host: As we finish up here, what do you see on the horizon in your field? And what are the big challenges that we want the next generation of best and brightest minds to be working on? What advice would you give to young aspiring researchers who might want to help tackle these problems?

Ashish Kapoor: One thing that I encourage everyone who aspires to work in this field, is to actually think about technology that would influence a large amount of people in the world, right. Often, you know, the technological innovations, by the time they trickle down to the masses, it takes ages. And given the power of AI, we want these technologies to be effective as soon as possible. Like, you probably know for instance, from the farming perspective, we need to double the yield of our crops by 2050 without increase in arable land. So, if we invest in AI technologies that can help solve that goal, we can assure that we will reach masses. So, I would strongly encourage folks to think about these problem areas where most of the population of the world gets positively influenced.

Host: Ashish Kapoor, it’s been absolutely fascinating talking to you today and I’m so grateful that you came in and sat down in the podcast with us.

Ashish Kapoor: Thank you. It’s been wonderful talking to you.

Host: To learn more about Dr. Ashish Kapoor, and the machine learning approach to building autonomous and robotic systems, visit Microsoft.com/research.