Playing With Reality - Episode 5

 

Sorry, this content can only be visible if Functional Cookies are accepted. Please go to the Cookie Settings and change your preferences.

 

Subscribe Now:

Episode Transcript

Menno Van Doorn: How close can a computer simulation come to real life? It's a question that the pioneers of the metaverse have been asking themselves since its inception. One of the most interesting technologies attempting to do this are called digital twins. These are mirror worlds which not only play with reality, they attempt to recreate it. This week we're speaking with Professor Karen Willcox, who specializes in digital twins, and Timoni West, who's working for Unity, the world's leading platform for real-time content creation. Welcome back to Playing With Reality with me, Menno Van Doorn. A new podcast series from Sogeti, the home for technology talent..

[music]

Of all the things we have talked about in this season of Playing With Reality, digital twins are probably the most tangible, results-oriented technology of the lot. Digital twins are best thought of as virtual representations of real-world systems. Think: a working digital version of a car's engine used to assess the real one's safety. Their use cases are broad and varied. From automotive to aerospace, healthcare, and entertainment. They could even recreate the human body itself. Given our focus, this season is all things to do with the metaverse, this is where I wanted to begin with digital twins.

I got in touch with some friends of ours at Unity. That company whose cross-platform game engine builds real-time 3D projects for various industries. I was lucky enough to speak with their vice president of emerging tools, Timoni West. Timoni has been with Unity for over seven years. Over that time, she has worked to lead advanced product development for spatial computing tools. Now she spends her time focusing on ensuring that the tools that Unity creates are available to all. I started our conversation off, as I always like to, asking Timoni about her views on digital twins and the metaverse.

Timoni, you're working for Unity. We could call it an extended reality tech company AR/VR augmentation simulation, whatever. Let's call it the metaverse for this moment or maybe we could just call it the hype. Is it beneficial that so many people are now talking about this metaverse or is it just words and is it now actions?

Timoni West: I think the metaverse is similar to the first words we use for any new branch of technology that will probably evolve and be called something else as the dust settles and becomes more ubiquitous. It is indicative of a real change in how humans interact with computers. They're becoming more ubiquitous, they are much more powerful. They enable us to have presence not only over Zoom, but real-time editing, Figma, or real-time presence in virtual reality. They are more world aware. They're running machine learning. There's a lot of meaty things happening with computers in this era. I think it's hard to summarize all of that. I think the metaverse is the word people are using as a placeholder.

Menno: Yes, great but one of the reasons why I wanted to ask you about this current hype, in words, is that from my perspective, I think we've bought the hype for quite some years ago, for instance, our company 10 years ago, was already doing a lot of things in the realm of digital twins. Could you tell us what a digital twin is and how it differs from let's say, an avatar or something?

Timoni: Sure. Well, after clicking a digital twin depending on the industry and the use case, I think digital twin is a loaded term insofar as for specific industry that has a very clear, precise definition. For other industries or general use, it has almost no definition at all. For industries that have used it for years, for example, manufacturing, and specific companies, when they say digital twin, they have a specific, almost file format or data structure in place. I think that's less common, I think now when people talk about digital twin use cases, the most common thread you'll see throughout any of them, is there are some time together of the digital and the virtual and that's really the key difference.

You can go into virtual reality, for example, and you can have a grand old time in a fantasy world that has nothing to do with actual reality. Digital twins by nature have either a relationship to some real-world object, or they reflect something about the real-world condition, or they're simulating the real world. There's always, that is what differentiates a digital twin from any other digital property.

Menno: Would you agree to say that we have passed the hype, talking about digital twins in an industrial context, creating value for many, many years?

Timoni: I hope so because when the hype cycle is over is usually when the real work begins. In any trend, there is a very sharp hockey stick curve up towards popularity and ubiquity, and then it very quickly falls off the other side as people get disillusioned by the tech. Then when it comes back, usually, that's when it starts to actually become part of people's lives. I think that's where we're at. Digital twins had a heyday about five years ago but what we see at Unity is that everyday companies come to us wanting to organize your data, make sense of it, be able to learn from it, be able to make visualizations of it. This is the real work beginning of actually incorporating digital twins into their workflows.

Menno: Maybe to the learning aspect, what can we learn from them?

Timoni: This is the part where I think it's incredibly exciting. When you create an accurate reconstruction of any given real-world system, you can then start to run scenarios on top of it. First, you emulate the behavior of the environment, or the object, or the machine. You can just run a million different scenarios on top of it and learn about what-- Basically, you can start to predict the future or you can start to figure out where things have gone wrong and what you need to fix.

That is really the moment where it goes from just being a nice visual representation of what's happening in the moment to actually something that you can learn from, and is a real tool for forecasting for making better products. That's the part where I think digital twins get extremely interesting and very high value.

Menno: You said, okay, an avatar could also be a digital twin. What do you mean with that?

Timoni: Well, there's a couple of different layers where it could have fidelity. For example, at the lowest layer, if you just wanted to have autonomous, non-player character types that are just simulating or emulating, say crowd movement in a space, that would be one example. It's an avatar, but it's not really an avatar of anything other than a generic [crosstalk] passenger. Then as you move up the ranks, you can get to something that is mimicking a person or be able to react, to respond to a person, you could argue on some level that even things like face recognition and voice recognition. They're not physical visual avatars, but they are actually an avatar component of your physical being.

Menno: When I started thinking about digital twins for this episode, I thought we would be talking solely about simulations, about creating digital versions of systems and processes so that automotive or aerospace engineers can better predict what is going to happen in their machines. It turns out, digital twins can be so much more. The fidelity to realize that they have means we can view them as an avatar, as the embodiment of a real person, even as we'll come to later, a replica that can be used in film, TV, and other forms of entertainment.

The scope of this technology seems to have no limits but let's bring it back to that essence. The more scientific use cases. I wanted to speak to someone who has been working with digital twins in a more academic sense, someone who has those scientific use cases in mind. I got in touch with Karen Willcox. Hello, Karen.

Karen Willcox: Hi, Menno.

Menno: Hey.

Karen: Nice to be here with you.

Menno: Karen is the current director of the Oden Institute for computational engineering and sciences at the University of Texas at Austin. Before this, she spent 17 years as a professor at the Massachusetts Institute of Technology. Her research is fascinating. She produces scalable computational methods for design of next-generation engineered systems. Digital twins in another name. She is currently working on projects for institutions as prestigious as the US Air Force Office of Scientific Research, amongst many others.

I wanted to ask Karen about some of the early incarnations of digital twins and given her background in aerospace engineering. I started with one of the earliest champions, NASA. I think the word digital twins for the first time was used by, I believe by NASA. Actually, they coined the word, digital twins. Maybe that's a good start for our conversation. Can you tell me maybe what that twin was doing for NASA?

Karen: Well, you're right, Menno. The term was coined by some work by NASA and the US Air Force in around 2010. That work was really focused on structural health monitoring and the role of digital twins in representing how the structural health of something like an aircraft would change over its lifetime as the aircraft is flowing and starts to degrade and is maintained. Again, that was 2010, so it's about 12 years ago, but many people often point much further back in time.

They point back to the Apollo program in the '60s and the '70s as NASA keeping what we now call digital twins but keeping digital simulators on the ground in Houston to track along with physical spacecraft going up into flight. There's a wonderful blog post by someone from Siemens that talks about the Apollo 13 crisis and the role that a digital twin played there. I think we can look at the phrase in 2010, but maybe the actual implementation by NASA many decades ago.

Menno: Yes. Well, you talk about Apollo missions-- We know it's ultimately important that these things are healthy, I would say. Even before they use these words, digital twins, what made it so important for them?

Karen: If we think about what is a digital twin? It's a representation in the virtual world. A computational model or a set of models that is representing a physical system and not just representing that physical system in a static way, but really being dynamic and changing so that as the physical system changes, the digital twin is reflecting those changes. How is this possible? Well, I mentioned the models. There's the computational models, the mathematics, and the physics that let us represent these systems. Of course, there's also the data, the sensors that are on the aircraft or the measurements of the human patient or the sensors that are on the spacecraft and that data are feeding into the digital twin.

That's what's letting the digital twin follow. Now you ask why is it important? It's important, because if you have a really good computational model, a virtual model that represents the system now and lets you predict what might happen in the future, then you can make better decisions.

You can make decisions. For Apollo 13, it was the decisions about how to bring the astronauts back home safely when this crop was damaged. For an aircraft wing, it might be a decision about how to fly or win, or whether to do extra maintenance. For us as a human, it might be a decision about some treatment for a disease or for our health.

Menno: Basically what you're saying is that digital twins can be life-saving.

Karen: Absolutely. They can be life-saving. They can make our systems safer, more efficient, more cost-effective, lots of potential benefits.

Menno: Now you talk about digital twins in respect to prediction. I think it is easy to understand why that is important for decision-making, but you also talk about a personalized future when you talk about digital twins. Well, what is a personalized future and how do these digital twins play a role in that?

Karen: When I think about what is different for a digital twin compared to a computational model that an engineer may have been using for many decades, that personalized part of it, that's a big part of the difference. To give you an example, aerospace engineers, I'm an aerospace engineer. Aerospace engineers have been using computational models to help design aircraft for many decades. How is a digital twin different from that?

There's personalization that says it's not a generic model, but it's a model that really is following the individual aircraft. It's personalized. The reason it's personalized is because it has this dynamic interaction with the data that we can collect from that individual aircraft to personalize the model and then to let it be dynamically evolving. That personalization really changes the way that we can use our computational models to think about decision-making in a way that, like we just talked about, can lead to a lot of benefits.

Menno: Is it personalized in the sense that this is a personalized plane, a specific plane, that one plane, or is it a whole family of the same kind of planes?

Karen: That's a great question, and the answer is yes, both. [chuckles]

Menno: Just wondering. [laughs]

Karen: You imagine a future where we would have a digital twin for every aircraft in your fleet. Maybe a good future scenario to think about is the day when we have autonomous vehicles helping with cargo delivery. Think about Amazon for example, and one day maybe we'll have autonomous vehicles helping to deliver Amazon packages in a way that again, has societal benefits such as reduced environmental impact.

We would imagine a digital twin for every aircraft in that fleet, and there would be potentially thousands, tens of thousands, hundreds of thousands of these vehicles. Digital twin for each one. At the same time, we'd want to have connections among those vehicles so that, if you may know, I don't know, you have a fleet of a thousand of one particular quadrotor and you start to see problems with a few of them. You can take that data and take that learning and get out in front of any problems you might have with other ones.

[music]

Menno: A digital twin then can function as a kind of common asset in basically any realm, by being able to simulate a highly personalized and specific system, it's easy to then assess the effectiveness of multiple machines as well. It's clear the life-saving opportunities this could offer. This technical side of digital twins is fascinating. I wanted to find out more about how they are actually put together. What team do you need to complete them? I went back to Timoni to find out because her team at Unity is composed of a fast array of talented individuals who come from all backgrounds. Do people want to work for your company? Are you always looking for new talent? How's the market?

Timoni: Oh, we are, yes. Let me be clear. A lot of the companies, let's use automotive as an example. Automotive car companies have artists on staff, they have designers, they have people whose job it is to be designing digitally and creating the car models. The thing is though, the file types that they end up with are not necessarily the file types that are very performance. They tend to be very heavy. They're very precise.

They're going to be used for manufacturing so they have to be as precise as possible. What Unity does is it allows you to transform a model of a car. The shell of the car, the wheels, each individual screw, and then use Pixie, which is our data transformation service, to take that, whatever file you have it in, and turn it into a lower res mesh that you can then visualize on a phone, visualize on the web, visualize on a lower powered computer, visualize in virtual reality.

There's a process there where you've got really the canonical item then you have the ability to transform it, but save all the data associated with it and then be able to output it out to all the different formats you'd want to use. Unity doesn't, or we can, we actually have a whole studio, but we're not the ones who are likely to be making the simulation for you. What we do is making the tools easier for you and any other company to be able to make digital twins and learn from them.

It's interesting too because there is a blurry line between emulating a digital twin object or something like a car, which is discreet. Then also emulating the entire world around it, which we tend to refer to as more in the realm of simulation. The two go hand in hand. Then there's another layer on top of that, especially for things like self-driving cars where you can emulate the sensors used and the machine learning underneath that has object recognition or edge detection, or street detection. You've got multiple layers of simulation going on. You have the base object, the environment they live in, all the scenarios you can run on top of that. Then emulating the machine learning layer and the sensors about that.

Menno: Okay. That's clear. I was just wondering if everyone is building these simulated environments. This goes back to maybe the origin of the meta- the idea of the metaverse that in the end, we will have an exact copy of everything somehow or some way. Do you believe in that futuristic vision?

Timoni: It's like when we talked about a digital twin of yourself earlier, I think, it will happen, but not in a way that we expect. I'm not expecting to open up a canonical file that is an exact one-to-one replica of everything at every event in the world. I do think clearly we're moving towards a world where humans really like to change the world. We like to change the earth as we see fit. I think a large part of the next wave of humans interacting with the earth is just recording everything about it. Really we do so we can learn from it. It has its downsides like creepy marketing, but it has its upsides. If we hadn't been so obsessed about keeping track of the information, we probably wouldn't know about global warming today, for example.

Menno: There was this situation that Bruce Willis supposedly had said that he was going to be licensing his own digital twin for future movies. Now we understand he's denying it. It's maybe fake news, I don't know. What would you make of these, first of all, about the idea that an actor actually could license his own doppelganger?

Timoni: I think it's really fascinating because we have had examples of formally, diseased actors who have been brought back to life, or performers who've been 'brought back to life' by the magic of digital animation. It's extremely interesting that we have a point where we have these actors who are well known for how they look, how they sound, how they behave, even across different roles, for example, in Bruce Willis is certainly one of those where he could just-- He's incapacitated due to illness. He could simply say, "I'm allowing my likeness to be used for performances, and I'm turning this over to the animators who actually do the work."

A lot of the recreated CGI performances, even with actors that are being scanned and their every move is tracked. Animators actually do go back and change a lot of those performances.

They don't match it one-to-one necessarily with what the actor does. They match it to what the director wants. There already is a disconnect between the actor's performance and what ends up in the final product. That was very surprising to me. I assume they would stay true to the way the actor had acted it out. That's not the case. They just go with whatever plays well on screen.

[background music]

Menno: The variety of different spaces in which digital twins are being used is quite overwhelming from global warming to recreating actors on screen. Their function can be to answer the ground questions of the world today, but also things more personalized like the health of human beings. I asked Karen about her research in personalized healthcare and how this might be utilized by companies who want to get into this space as well. You are doing, at the university, a lot of interesting research on all different kinds of digital twins. Can you give maybe one or two nice examples of what the things are that you are doing research on?

Karen: I mentioned earlier I'm an aerospace engineer, so I'm trained to think about airplanes and spacecraft. In our recent work, we're collaborating with the Center for Computational Oncology here at UT Austin to build digital twins for cancer patients. This is really a collaboration that came out of that early work I mentioned where we tried to think very foundationally about what is a digital twin and how do you represent it mathematically.

In talking with the oncology experts, the folks who live and breathe and think about cancer, it's just incredible to see the analogies. We talked earlier when you asked me about what is structural health of an aircraft. I gave you an example, an analogy with a human. Seeing the analogy, while the physics is very different, the challenges of building a digital twin for a cancer patient and building a digital twin for an aircraft's structural health, there are a lot of common challenges.

We've been working with the folks here and trying to take the methods that we've developed and bring them over to representing a cancer patient digital twin. Again, it's just been absolutely fascinating. I've learned so much. Also just to see, here's an example where a digital twin technology- the technology doesn't have to be perfect before it could really have an impact and change outcomes for people.

Menno: This fundamental research done at universities, at your university is important for organizations, companies to take it further. I think a company can do this kind of fundamental research in building that kind of model or can it?

Karen: It's a great point. Something like a digital twin has many needs when it comes to research. Some of the needs are at this foundational level really digging into the mathematics and the algorithms and the computations. There's also needs across the spectrum, all the way out to the implementation in real systems and understanding how digital twins might interact with human decision-makers. There's just so many facets.

I think you're absolutely right. Some of this work is best suited to maybe more of the basic research in the university. Some of it is really suited to industrial R&D. Then of course the partnerships. This is a big part of what we do here at the university. Even though we're engaged in basic research, we find that partnerships with companies, and these companies include industrial end users of digital twins, but also software companies who are developing the software and the methods, the models that will enable digital twins.

Those partnerships are so important. They help us to have impact with our work, but also they help us to understand what the real-world gaps are and how we can contribute.

Menno: I love that you used the word that digital twins have needs or the research on digital twins like it's your baby or something. It's human.

[laughter]

Karen: My research is my baby and my Students and postdocs, they're my family. They're my academic family.

[background music]

Menno: I love this idea that something like research into simulating cancer can be used by companies to further their own goals. It's this kind of cross-pollination of ideas that is so vital for people working in the tech space. It's something that we have seen time and time again across this series, but now in the future, the use cases for digital twins right now are fascinating enough. What about in 10 or maybe 20 years' time?

Where is it all going next? I went back to Timoni to find out. She thinks that democratizing this technology, so it can be used by all, will be vital to its growth. Is there a general idea at Unity of where digital twins could be used in the future, but not yet? This point on the horizon ideas about this technology.

Timoni: I think we're there. The amount of people that come to us, the amount of digital information that exists today often just needs to be organized. If you name a major company we've done work with them or are actively partnering just automotive and manufacturing and film and obviously games, of course as our bread and butter. When it comes to digital twins, I think it's happening. It's happening today. What I think we lack, and this is really what my team is focused on, is making the tools more accessible, making them easier to use, making it so you don't need to hire effectively a gameplay engineer to make a simulation.

Unlocking that data, making it easy to bring in large complicated file formats and being able to easily transform them into something you can see and reason with. That's really 100% where our focus truly lies. It's digital asset management. It's being able to edit things easily. It's being able to distribute them as easily as possible.

Menno: If there is a consumer metaverse, one of the things people talk about for so many years already is my body double will go shopping online because it's an exact copy of me, I will never have to have the trouble of not fitting the suit that I've bought. That's the simple use case. My only question is why isn't it here yet? What's stopping us? Is it you? Can't you handle the technology? Do you need help?

Timoni: [laughs] Actually no. Unity could do that. It's funny you mentioned that one because that specifically came up in one of my favorite articles that I quote endlessly, "Will the information SuperHighway be the death of online or death of retail?" It's from I think 1994 Fortune Magazine. In it they specifically mentioned you'll shop, they said on your television, you'll use your remote, and they said specifically because the creator has your biometric information, it can just tailor the suit to fit you perfectly. The clothes will always fit you perfectly.

Menno: Voila

Timoni: The reality is we've had that technology for a really long time. The problem is not getting the data about you to anyone. The problem is there are very few supply chains in the world set up to do custom clothing. The problem is on the other side. That's actually where you would need to have the innovation.

Menno: Yes. Maybe this leads to my question about the unknowns. When we talk about the future, let's call it the future of metaverse, digital twins, social media. This is a difficult question. What could be something that's in your mind somewhere, a question or something that could be important for the future development of use of these technologies that isn't talked about that much, that could surprise a lot of people?

Timoni: One interesting trend which may or may not be on people's minds is that there's obviously a rise in digitizing a lot of civic infrastructure. There is actually work being done now to take software that was developed for one city and open sources so that other cities can use it as well which I think is awesome and really an interesting way of cross-city pollination to-- Not that I think that things necessarily have to be the same in every city or standardized, but just cities really helping each other out to be able to make better decisions and have better infrastructure for their cities. I don't know if many people know about that, but I think it's really cool

[background music]

Menno: This idea of digital twins functioning in real cities and real spaces is something that both of my guests today kept coming back to. It's diffusing the digital and the real on a vast scale and could provide solutions to problems which urban planners and governments have been grappling with for centuries. Do these digital models have the capability to do so to change the very places we live in? Well, Karen seems to think so. When we look at the future, and this is the fun part, I think of our conversation. What would you hope to have achieved in 20 years looking back that could be a really interesting breakthrough in the work that you are doing, that you would be proud of, or it could be something that is important or that will bring you the Nobel Prize?

Karen: I imagine a world where every time an engineer or a team of engineers sits down to design a complex system, at the same time that they design the physical system, they're also designing the digital twin. What does that mean? That means that we are thinking about a complex system and pick your favorite one, an aircraft, a bridge, a building. We're not thinking about the design as a one-time deal and then we are deploying the system and then it becomes somebody else's problem to maintain and operate but we're really thinking about the full life cycle of that system and architecting the digital twin in the same way we architect the physical system.

Why? So that now we are building up the digital and the physical together and that digital really will follow that system through its life cycle. Just to give one concrete example, if we really want to take on environmental impact and think about environmental impact over the life cycle of transportation, of urban infrastructure, you're moving towards the idea of a smart city, we are going to need all these physical objects to have those virtual representations so we can understand how they are changing and what that means for our decision making. I don't know, I guess that's an engineer's fantasy. Maybe it doesn't get-

[laughter]

- too many people excited, but my fantasy is that every engineering system would come together with its digital twin.

Menno: Well, I'm happy that you have this fantasy and we need more engineers to make these things. I live in Utrecht in the Netherlands, in a very nice city. I can imagine that in 20 years there will be a digital twin of my city. What could that bring to me or to the people that live in Utrecht? What things can we think of that could be important to have a digital twin city for instance?

Karen: That's a great question. I'm imagining now Menno your day in, I don't know how many years, 20 years, 30 years, 40 years, where you wake up in the morning, some data is collected from your body in whatever way, your own digital twin is updated and your digital twin suggests what you might have for breakfast or perhaps what you might not have for breakfast. The digital twin of the city is optimally managing the way that you are going to commute to your job by also looking at your health data, looking at the weather, looking at traffic conditions, is really providing dynamic information to you to help you make better decisions and factoring in all these different aspects from health, wellness, environmental impact, things like congestion.

That sounds a little far-fetched but I think many of the pieces of that puzzle we are not too far away from. You asked specifically about cities, thinking about how a smart city, how a building could be optimized to recognize when there are and are not people in there to manage the energy footprint of the building. This is already a reality in some places around the world. Really just thinking about how that is done at scale and is integrated, I think is really exciting.

Menno: Normally, we end this show by saying, next time, let's meet in a virtual space, we call it the metaverse but maybe we should imagine you and me meeting each other maybe in 20 years, both physically and digitally with a twin on our sides and see how that conversation would go.

Karen: [laughs] I like that idea.

Menno: Thank you so much, Karen.

Karen: Thank you, Menno.

Menno: If there's one word that I have taken from my conversations today, it is integration. Highly specialized digital models will allow us to integrate processes and find solutions to technical problems. This could soon be groundbreaking. Karen's idea of a world where you wake up and are given dynamic real-time data of how you should get to work or eat breakfast is a testament to just how revolutionary this technology could be, moving from simulation to reality.

As these digital twins start to outsmart human beings, the question remains whether we are smart enough to listen to their advice. In certain situations we will for sure, but do we really want to live in a city that's too smart? If the answer is no, will our own stupidity stand in the way of a more sustainable planet? What started with a discussion on digital twins ended for me with these existential questions. It shows that the metaverse is indeed a mirror world and when we look into that mirror, we see ourselves, our hopes, and our fears instead of technology.

That's all for today. Thanks so much for listening and a big thank you for both our guests, Timoni and Karen, for their introduction into Digital Twins. If you enjoyed this episode and want to let us know, please do get in touch on LinkedIn, Twitter, or Instagram. You can find us at Sogeti and don't forget to subscribe and review Playing With Reality on your favorite podcast app as it really helps others to find our show. Next week we'll be taking a look at one of the biggest industries when it comes to the metaverse as we explore the cultural impact of gaming across the globe. Do join us again next time on Playing With Reality.

[music]

[00:37:09] [END OF AUDIO]