VFX in VR: Storytelling and Innovation with LightSail VR's Matthew Celia

Discover the fascinating world of virtual reality storytelling with our special guest, Matthew Celia, the creative director at LightSail VR. Through an engaging conversation, we promise you’ll gain unique insights into how Matthew’s journey from film school to pioneering VR storytelling has shaped the narrative landscape, where story takes precedence over technology. We highlight the remarkable work on projects like "Paranormal Activity: The Ghost Dimension," illustrating the synergy between technical innovation and compelling storytelling that defines LightSail VR’s reputation.

Explore the evolution, challenges, and triumphs of VR storytelling as we unpack the lessons learned from early VR camera placement errors and the transition to 180-degree formats, which elevate viewer engagement. By simulating the experience of a live concert, VR storytelling grants audiences the power to engage actively with narratives, commanding their complete attention. We praise the VR series "The Faceless Lady" for its captivating performances, which showcase the unmatched power of acting to enhance the immersive experience in a virtual world.

Venture into the technical heart of VR production as we tackle the intricacies of sound, image, and VFX. Hear how spatial sound and stereoscopic vision work hand in hand with cutting-edge tools like the Canon Dual Fisheye lens to bring depth and precision to VR experiences. We address the VFX challenges uniquely posed by VR, where every shot demands meticulous planning and correction. Drawing inspiration from classic cinema, we discuss the art of mastering camera movement in VR, ensuring comfort while maintaining engagement for the audience. Join us on this journey as we celebrate the thrilling intersection of VR and entertainment.

Transcript

Paul DeNigris: 0:00

Step inside Kilok Castle, a magnificent stone fortress deep in the Irish countryside and home to Lady Margaret, a tragic spirit doomed to walk the crumbling manor for eternity. Three couples compete to survive the ghost's sadistic games in The Faceless Lady, an immersive stereo VR experience from Meta, Crypt TV, and Eli Roth. This week on VFX for Indies. Welcome to VFX for Indies, a podcast about the intersection of visual effects and independent film. I'm your host, Paul DeNigris, visual effects artist, filmmaker, and CEO of boutique VFX shop, Foxtrot X-Ray. Joining me on this episode today is Matthew Celia, creative director and managing partner at LightSail VR, A bleeding edge studio committed to pushing the envelope in terms of storytelling in virtual reality. Welcome to the podcast, Matt.

Matt Celia: 1:07

Hey, thanks for having me, Paul. Good to be here.

Paul DeNigris: 1:10

So on this podcast, our goal is to educate filmmakers, usually independent filmmakers who aren't super technical about aspects of visual effects. And in this case, virtual reality. So if you like what we're doing here, viewers, uh, if you like what we're doing here, please like, and subscribe. So you don't miss any of our new episodes and you can check out our back catalog on VFX for indies. com. My company Foxtrot X-Ray had the opportunity to collaborate with Matt and the team at LightSail VR. On The Faceless Lady, which I teased at the top of the show. We'll get to that in a second before we do Matt, just tell us a little bit about yourself and your career. It seems like you've been in the VR space for a long time.

Matt Celia: 1:49

Yeah. So thanks. Um, I'm a, I'm a film school kid, you know, went to film school back in 2003 Chapman University. That's where I met my now business partner, Robert Watts. Um, and you know, always was interested in directing and creating content, but always had a kind of a technical edge. When I graduated, I went right into like post production, working for Warner Brothers, doing digital first content. I ended up doing some commercials, working at a commercial production company for a while. And, uh, eventually landed into creating my own documentary, shopping that around, and, uh, getting that hard reality of struggling to break into the 2D, very competitive landscape. Um, at that time, Robert, my business partner, he, handed me a piece of cardboard with a phone inside it and said, Hey, I think this is like 2015. He's like, Hey, I think this is going to be big one day. You should, you're technical. You should take a look. And you know, like creatively, I've always had a artistic theater style to my stuff. It's very real in the moment. It doesn't really rely on a lot of overt visual effects, I would say. I mean, there's always like a lot of hidden stuff, but. Yeah. You know, I like to be in the moment and be in the real present. And when I saw the piece was called Walking New York. Uh, it was by Chris Milk's company at the time. And it was just like, I was in New York and I was in a helicopter and it was just letting the scene play out and it felt like theater and it really ignited a passion in the kind of storytelling that I wanted to do. So I spent the next, like, 48 hours watching furiously everything I could find. And I went down to VRLA, which is a big conference here for VR. And it was a bunch of hacked science fair projects. And everything I saw was super interesting. But what I noticed was everything I saw was very much about technology. And not about storytelling. It was very much like, Hey, I put a camera down, isn't that cool? But there wasn't that sense of character, that sense of conflict, the sense of emotional stakes that you get in a good story. So I called Robert up and I said, Hey, I think as an artist, someone who's been struggling to find how they fit into this world, I have something to say finally. And I think this is it. So we bought like 10 GoPros when we started making content and people saw us on social media and. We ended up creating one of the first pieces for Paramount Pictures called, uh, Paranormal Activity The Ghost Dimension. It was a in universe original content written by the writers of The Ghost Dimension that, uh, tracks like these three kids summoning a ghost in a garage. And it was one of the first VR pieces that we were That had real people acting scared had a mechanism that was, you know, in universe, you know, we used all the paranormal activity like blue and the fuzzy cameras fitzing out like really dove into the world and the rules of the world. And it resonated with folks. Uh, got 10 million views. Um, and we, we would walk around at conferences with like, uh, Gear VR in our backpack and like hand it to people and like they'd watch this horror content. It was so wild. Um, but they were like, no, that was cool. I feel like I'm emotionally gripped by it. And that was the thing for us that said like, Hey, we can take the a hundred years of storytelling and put it into this new medium and we can start to figure out how to translate that. And that is the like ethos of LightSail VR is we're story first people. We're tech heads. We're nerds. Um, but at the end of the day, we work with directors and creatives who want to get into the virtual reality world and experiment with this new medium. And we translate how that actually works. Um, and that's been what we've been doing since 2015. So almost 10 years now, it's a pretty wild ride.

Paul DeNigris: 5:51

That's great. Yeah. I, you know, that was actually, my next question was going to be about this phrase story first, cause it's all over your social media and your website. Like it's, it's your mantra. Obviously

Matt Celia: 6:01

it's, it's, it's a finding aspect of what we do. And, you know, Robert and I. We could do a lot of stuff and we've done a ton of stuff. We've been very flexible as a company. You know, we've done everything from training to, to, uh, entertainment, like The Faceless Lady, but our heart is in entertainment and our heart is in storytelling and the unique thing that I think LightSail offers to a lot of productions is that we speak both sides of that. That chasm. We are very technical, but we also understand like the needs of creating characters that emotionally grip you and that puts you there. And our phrase here is that immersive is not just about the physical bounds of reality. It's an emotional immersion as well. And you can't forget that because you can be immersed in a 2D film as much as you can be immersed in a VR project. It's about how good your story is.

Paul DeNigris: 6:50

So what are some of the things about VR that you think really lend themselves to, to storytelling? You know, what, what is it, what is that secret sauce that you, that you guys have discovered that makes stories compelling in the VR space?

Matt Celia: 7:02

Yeah, if I had to distill it down to one word, I think that word would be presence. It's the idea of feeling like you're actually there. Um, and if you start to think of all your creative decisions, As subservient to that word, and that's your North Star, good things, I think, start to happen. So, it's about feeling like you're standing in the best spot in the room, where you would naturally be. A lot of early VR creators would put the camera in the dead middle of the room, thinking like, Oh, that way I can see everything all around me, because 360 at the time. And if you actually go to a party and you look at where people stand, Nobody stands in the middle of a room. Like, that just doesn't happen. And so understanding, like, where a good vantage point is helps feed that reality. Like, how high a camera is. That feeds that reality. How people interact with the camera. Feeds that reality. And so, the idea is to make people feel like they are there because the greatest advantage of being in VR Is that you look around and you are in the story and you want to give the audience that agency, that choice to be able to choose. I'm going to look at the painting on the wall over here. I'm going to look at, you know, this character over here because that choice. There's something, I don't know, there's some neurological thing about the way we are built as human beings is where when you make that choice, when you initiate that action, the connection you have to the material is somehow stronger, and this is backed up by dozens and dozens of studies in this field, and it's why virtual reality is such a powerful training tool and a powerful, um, like therapy tool, um, is because there is something very real about you saying, I looked at that, I looked at that. That makes it a more active viewing experience rather than when you watch TV on your couch and you veg out, it's very passive and things just fly by you. That's okay too. But VR, I think naturally asks you to lean in a little bit.

Paul DeNigris: 9:09

Yeah. I mean, it sounds like in a way you're giving a little bit of directorial control to the viewer, kind of making them a partner, right? You're, you're deciding this is the best vantage point, but then the, the viewer ultimately has the choice of what they're looking at, right? Yeah.

Matt Celia: 9:23

And, and, you know, VR has gone through, um, VR is a big topic, right? There's interactive VR, there's passive VR. When we started, we were doing everything in 360 degrees. And that was like. People really had agency because you could look behind you. You could look anywhere and directing that was a challenge because you really had to think a lot about, all right, what are people going to do? How are we going to get people to look at the thing I want them to look at? Now we've kind of evolved to more of a 180 format for a lot of narrative content, which I think is. the best medium of this. And you sacrifice a little bit of that immersion because obviously when you look to your left, you see like a black line that goes into nothingness. But you know, as a creative, as a storyteller, I'm still saying, Hey, generally look this way. I'm generally framing my shots. I can cut faster. I know that you're going to be looking that way, but there's still such a wide canvas that you still have that agency That it's the same kind of agency you have when you watch a concert or a theater show. And you know, it's like that same kind of reason, like when you watch a concert on television and they're cutting shot to shot to shot, it's cool. That's one kind of the experience, but would anyone say that that replaces the value of going to a live concert? I don't think so. And I think it's because at a live concert, you're making that conscious choice to look at the guitar player, look at the drummer, look at the singer. Um, and it, it creates that same kind of agency.

Paul DeNigris: 11:00

Uh, although most people are, most people are looking at their phones. That's true. Which is a whole different thing. That's one of the nice things about VR is it's hard to look at your phone when you're, when you're consuming VR.

Matt Celia: 11:12

I mean, I think it's a great medium for, for creatives. Yeah. Like. Think about it. Right now, a lot of directors, they create content for phones or for, for TV even, and they're watched in the worst circumstances on earth. Like, there's people talking, there's texting, you're like making dinner, my wife's folding laundry or doing whatever, like I'm trying to watch something. When you're in VR, everything's gone. The only thing that you are looking at is the stuff that the creator put in front of you. And that is tremendously powerful, but at the same time, I think that's a really big responsibility because it does mean that. If you can't keep people's engagement like that, then you have serious story problems.

Paul DeNigris: 11:57

Yeah, for sure. And a lot of it, I think also comes down to the acting, right? The acting has to be, has to be engaging and naturalistic. There's a, there's a moment in, uh, in The Faceless Lady that totally sold me on the, on it as a show. Uh, it's when, um, the, the, the, the lead character Ella, right. It's when Ella is sitting on her bed. In the room and she's she has this this emotional moment. She's crying and we are we are sitting like as if we are, you know, kneeling on the floor in front of her like we would if if we were trying to comfort her in in that situation. Right? Yeah. And it's like, okay, I'm I'm I'm totally the In this world and on this journey with these folks. And then when, when some of the characters die and you, you guys put the camera like right on the chest of the dude who's dying and you watch him take his last breath in VR and it's like, okay, that's a, that's a real visceral experience. Yes, I can look all over the place, but I don't want to because the acting is so riveting and drawing me to it.

Matt Celia: 13:04

And I will say, like, John Ross, the director of The Faceless Lady, was, is just a master of moving the camera. And he really, we worked really closely together and prepped for that production. We did a bunch of camera tests. We did different heights. We did different distances. We did different angles. And we put it all in a headset to try to build the visual language of the show, to know how far we could push, what would work, what would not work. And a lot of those moves are the result of many, many takes to get it just perfectly dialed in, right? To feel comfortable in VR. Because there's this whole comfort feeling when you're moving the camera as well. And I just think he did a really good job of Of listening and watching references and thinking carefully about the vantage points and the feelings he wants to elicit out of audiences. Like there are a number of times in the show where he does something that does make you feel a little like, Oh, I don't know. Oh, that's a little weird. You know? And he used it not just because, Oh, this is a cool shot. He was very specific about it. He's like, no, I want someone to feel off balance. Their friend just died. We should be feeling off balance. And I love that he did that because I think that that is using the medium to its fullest potential.

Paul DeNigris: 14:20

Yeah. It's another dimension we don't have in, in 2d, uh, filmmaking at all. So you've mentioned a couple of times, you've said the, you've said 360, you've said 180, uh, you know, for, for the, the folks out there who maybe are not familiar with... super well versed in VR. Can you just talk like, you know, we've got 360, we've got 180, we've got mono, we've got stereo, we've got passive, we've got interactive. Could you just give us sort of the menu of, of the different, uh, options out there?

Matt Celia: 14:48

So VR, like when you say VR, like think of VR, like you think of television. It is a very big and vast open medium. Basically VR is anything consumed inside a virtual reality headset. Um, so the kind of content that LightSail is most known for is passive, linear, immersive video is kind of what I call it. Um, and immersive video comes in a variety of flavors. The first is 360, which is 360 degrees. And that is usually captured with many cameras looking every single direction. All around. And so you can look up, down, left, right, backwards, forwards, and you feel completely immersed. It's like Google Street View, but like, you know, a video. Um, those flavors also come in 2D, monoscopic, and 3D, stereoscopic formats. Um, so, 180. Is just the front half of that sphere, and typically 180 will most always be stereoscopic because it's easier to do, and it really adds the immersion. Um, but of course you can make it monoscopic formats as well, and when I say monoscopic and stereoscopic, that's like if you're watching, you know, a 3D movie, because like you're wearing a headset on your face, so you might as well take the advantage that you have a lens of two different eyes. And get that depth, because that depth can be really exciting for storytelling. And also, since you're working in a format that doesn't have depth of field. Right. You don't have shallow depth of field and like a blurry background playing with depth in a stereo sense is a really powerful storytelling tool to enable you to say, this is what I want to focus on. You know, this is the core of my scene.

Paul DeNigris: 16:37

Gotcha. So, so putting the most important thing sort of at that convergence point is a, is a term of phrase that we used a lot. Yeah.

Matt Celia: 16:46

Yeah, exactly. And we don't, you know, a typical flat. 3D movie will move that convergence point to be what screen space is because that's the most like the focal thing in VR It's slightly different because we do it more like how you see whereas your convergence point You're is way in the distance because you don't see stereo depth past like 30 40 feet And then everything comes out towards you. So you only get that. Um, I believe it's negative parallax come towards you. So you only really get that. Um, but you know, it's about placing your subjects in this like sweet spot zone between like three and 15 feet where things look particularly good and sharp and accurate. And that's like, kind of like in life. When I walk up to you in the real world, I'm not standing 20 feet away from you. I'm also not standing one foot from you. I'm like, you know, that three to like six feet

Paul DeNigris: 17:41

realm, you know? Right. If you're six inches from the person you're having a conversation with, it gets a little uncomfortable.

Matt Celia: 17:46

It gets a little uncomfortable.

Paul DeNigris: 17:47

Both in VR and in...

Matt Celia: 17:49

real life! And that goes back to, like, creatively, thinking of the camera like a person, and like, thinking about where is your presence in the space.

Paul DeNigris: 17:59

Uh, I imagine the use of sound is also really important to help direct the viewer's attention.

Matt Celia: 18:05

I mean, less than you would think, to be quite honest. Like, it's definitely a common thing to say, like, use sound to direct people's attention. But the reality is, is that, um, unless you back it up with visual, like, something visually that, like, Says, Oh, that is what I'm seeing. There's so much going on. There's so much sensory overload that it gets overlooked. And like, that's a really important point to bring up is that you have to understand when someone put this headset on their face, especially they're new to it, and they've got this enormous canvas to look at, and they've got this spatial sound and they've got the 3d, that is a lot of sensory information. And it can be absolutely overwhelming to be in that. Um, so you have to take it slow and you've kind of got to hit the nail on the head a little bit. Um, spatial sound is huge. It's that thing that adds to presence. And I would say it's almost more important in VR than it is in a flat film, because we always hear 360 degrees always. And so having that element of it really does. You know, tell your nervous system, Oh, I'm actually here.

Paul DeNigris: 19:19

Yeah, for sure. Uh, particularly like the battle scenes at the beginning of The Faceless Lady. Yeah. You're hearing this, this full. You know, world of these guys fighting and, you know, all of the stuff, the fire and the swords clanging and things like that. And it, uh, and it really puts you in it in a way that, yeah, watching, you know, with your soundbar under your television, as good as that may be, it's still not, you know, being beamed right into your ears.

Matt Celia: 19:52

And it's, um, it's head tracked, right. I think people don't always understand that it's actually, it's It's real spatial. So if I'm hearing a character talk it right in front of me and I turn my head to the left, now I'm hearing it just in this ear, as if they are still just right in front of me. There's actually like a spatial level. Anchoring of all the sound objects. And that's the thing that really elevates it and makes that feel like, Oh, I'm actually here.

Paul DeNigris: 20:20

Uh, so you mentioned monoscopic and stereoscopic, so one eye versus two eyes, right? So, uh, just really quickly for, for the non technical folks, the idea is we see, we see two slightly different images, you know, one in each eye. Normally, right? If you've got, we've got stereoscopic vision, so we're seeing a little bit of separation and that separation is what our brain turns into three dimensions and turns it to depth perception, right? So stereo, we're taking advantage of that. We've got a left eye image and a right eye image.

Matt Celia: 20:53

Right. And the cameras we use, the lens we use specifically on The Faceless Lady is the Canon dual fisheye, which really has a lens for this eye and a lens for this eye, and that's what's capturing onto a single sensor. So you do get about 60 millimeters worth of, uh, parallax on that. Yeah. Which is, which is the average for, for, It's an average for human, for human eyesight. Everyone's between like 55 and 72.

Paul DeNigris: 21:18

As opposed to something like, uh, like avatar where they were shooting with two complete camera rigs and this whole, you know, uh, complex contraption to put the two lenses to have two cameras and have one shooting down into a mirror so that they can create the separation and all that. And they can dial the separation.

Matt Celia: 21:37

You can shoot VR just like that. But the problem becomes when your field of view is 180 degrees up. Down left and right it, you would see the other camera system right above you, you know, and, and that becomes a huge rig and very complicated.

Paul DeNigris: 21:52

So, so that, that dual fisheye lens really has kind of democratized the process.

Matt Celia: 21:58

It's been the biggest, uh, game changer. I think when Canon showed up with that lens and it's not just a dual fisheye lens of like, it's a really good dual fisheye lens. Like it's an L series glass. It's. Amazingly sharp. It's got very, very low chromatic aberration. It's got low flare. It's got, um, very low distortion values. So it's, it's It's a very high quality precision instrument. And I think Canon did an incredible feat of engineering with it, to be honest.

Paul DeNigris: 22:36

Um, they certainly that's lens certainly makes the compositor's job harder because we can't, we can't, you know, throw a couple of pixels of blur on something to hide, hide the seams, you know, when everything in the frame is super sharp, um, you know, everything that we added to the frame had to be Equally sharp. And there was no, a lot of the cheats that we do, uh, in, in regular 2d compositing, um, yeah, VR is not available to us.

Matt Celia: 23:04

VR is just very unforgiving for visual effects. You know, also you're taking what you do and you're blowing it up like so large onto a screen. So, and, but when you're working on it, you're working in it on a small computer and it's like kind of tiny and it's, it's just, it's, it's the challenge.

Paul DeNigris: 23:24

Yeah. So, you know, for, for those who are not, not experienced with this, that dual fish eye is going to shoot two. Uh, two 4k images side by side and they are spherical, right? So they, they're not flat lines are not straight, right? The, the, the photo, the pictures hanging on, on Matt's wall behind him would all be curved. All of those lines would be curved, right? So now we have to, we're either working in that spherical space and making sure that everything we do is respecting that curve, or we're taking. A little section that we're going to work on and taking that curve away, straightening it out, doing our work, and then putting the curve back on and sticking it back in the frame. So it adds a lot of complexity and a lot of like these sort of geometric gymnastics that we have to do to get to get everything to sit right in the world. And then we're doing it in two eyes.

Matt Celia: 24:16

Right. So what you do has to be exactly the same on each eye, except for your horizontal disparity. Right? And that's that's a huge challenge because, you know, you're cutting the character out right with some roto. Well, you have to do the exact same roto on the other eye, and then you have to offset that, and then make sure that that's perfect, and if you don't, then you're going to have slightly different rotos. So like, we did a project where a visual effects team tried to convince us that they could use AI rotoscoping, and I said, it's not going to work. And they were like, yeah, it'll work, and they went ahead and did it, and it did not work, because the AI couldn't do it the exact same on each eye. So sure. The left eye looked fine and the right eye looked fine, but together they were slightly different and they did not look good.

Paul DeNigris: 25:04

Yeah. Yeah. We, we encountered a number of times where, uh, the Roto was the issue, right? Cause you've got, we'd have, we'd have, um, Ella's running at one point and she's running past the camera. And so the parallax was drastically different from one eye to the next. And the details of her hair were different from one eye to the other. And when we, the Roto, the Roto team gave us mats that were perfect, right? If you looked at left eye. Perfect. Look at the right eye. They were perfect. But then you put them together and it was like, something's, something's not right. Yeah. Something's wrong here. It doesn't feel right. You know, that was a note that you guys gave us a lot was this doesn't feel right stereoscopically. And it sort of drove me crazy because I'm like, I'm looking at, I'm like, This pixel lines up to this pixel and then it lines up to this pixel in this eye,

Matt Celia: 25:56

but then you look at it and there's like, there's that disparity and you're like, is it too far forward? Is it too far back? Like, and there's always this like little bit of like grid warping juice that we had to, like, we had to do, um, some, some roof replacements and like patches took for ever because the 3d roof was not a hundred percent like aligned with like the lens and we just had to like. Sub pixel warp it, which is challenging to do. Uh, on huge footage files.

Paul DeNigris: 26:28

Um, and you guys, uh, you guys work in, uh, resolve in fusion, correct?

Matt Celia: 26:35

We work in resolve, in fusion, um, primarily for financial reasons. Um, just because it's, it offers pretty much the whole tool set we need at a dramatically reduced price. But, you know, on The Faceless Lady, the thing that we explored that was really powerful was the integration with the actual timeline. Cause we didn't have to do the visual effects process of having to bake plates out and export plates and check those and do all that stuff. And artists could open up the timeline, select his or her clips, right click, go create fusion composition. And they had everything they needed in context right there. Um, and that was, that was pretty powerful. Um, saved us a little bit of time.

Paul DeNigris: 27:17

Yeah, that's gotta be really helpful. How is, uh, fusion in terms of. Being able to monitor your work in a headset in real time.

Matt Celia: 27:25

It's tough. It does have support to plug a VR headset into it. Um, but we've created a tool called vr.ndi at LightSail, which uses NDI to sling pixels and put it in a VR headset. And so because we're a distributed shop and everybody dials in from wherever in the world that they're working, all the computers in the closet have a plugin called Nobe display and Nobe display sends and eat NDI signal out of resolve. And we use NDI bridge, which is free by the way, to sling that back to the artist's home computer, where they have a headset, uh, where that home computer can receive that signal and then broadcast to their, their quest headset. So they're able to see stuff it's compressed. It's not always a hundred. Like if you're doing the sub pixel work, they downloaded it to their local machine and did that so they can have full res quality. Um, but if you're just placing stereoscopic titles and you're just like getting into the right space. It's enough.

Paul DeNigris: 28:25

Yeah, that's great. We, in my shop, we work in Nuke and Nuke has the ability to feed a VR headset, but it's not real time. And you can't see your mouse and you're in VR, you can't see your, you know, your mouse cursor or anything. So we, so it would be, you know, headset on. Right. And you set your perimeter. So you could lean forward and you'd be able to see through the world. Yeah. I'd be like. Lean forward, tweak something, lean back, wait for it to refresh.

Matt Celia: 28:53

Yeah, or you do, or you do that thing where you take your headset like on and off all the time. You're like, uh, what is this? What is this? Um, I use the, um, I use the Quest Pro when I'm at my desk and you can see like it doesn't have the bottom bit. And like, that's killer for me because I can see my keyboard and see my mouse.

Paul DeNigris: 29:09

Oh, that's, that's great. Yeah. For me, the, uh, the challenge was I wear glasses. Yes. So, so I would, that it was just killing the bridge of my nose, taking the head, the headset on and off. And then I got the, uh, Zenni eyewear does the, uh, we're not sponsored by Zenni, by the way, but Zenni, if you're listening, I would love a sponsorship. I love your glasses. Uh, Zenni has the, uh, the inserts for the, for the quest. Um, So I got those in my prescription and put those in and that was great. But then I couldn't do the on off because I'm totally blind with no glasses. So I'd have the headset on and could see clearly. And then I take the, and I'm looking at, you know, fuzz. So

Matt Celia: 29:46

I used to have glasses and, uh, it took working in VR for like seven or eight years for me to finally go like. I think I'm going to get the surgery. I think I'm going to do the LASIK and, uh, it was worth it.

Paul DeNigris: 30:00

Yeah, smart move. So in a typical show, um, and, and I know there's probably no such thing as a typical show because every show is kind of, but, but typically how, how is VFX fitting into your projects? You know, how are you using the effects?

Matt Celia: 30:19

Here's the, the, the reality of VR filmmaking is every single shot is a visual effect shot. Every single one. Because at the bare minimum, you're taking your fisheye lenses and you're applying, you know, an ST map or doing some sort of unwrapping to get it into equirectangular space and you're reconverging your stereo and so on and so forth. And then typically you'll have stuff that needs cleanup. Shadows, tripods, you know, like gack, whatever. Like there's, there's, uh, VR is a lot about what's invisible visual effects. And I think this has been talked about a lot, just in the broader visual effects community. Cause you have a lot of productions coming out saying like we did it all practically. There's no visual effects. And you're like, what are you talking about? There's like 2, 000 visual effects shots just to clean up like, you know, just to do a merge or a comp or take out a shadow or take out all this stuff. And so. Visual effects in VR is really about making the visible invisible, because again, going back to that word presence, you want to maintain the idea of presence. And if you're seeing a camera shadow right in front of you, because the sun's behind you when you shot the scene, well, that's pretty distracting because you're looking at a camera shadow instead of just the ground. Um, and so there's a lot of that. You know, I think we were really fortunate on The Faceless Lady to have the opportunity to collaborate with you folks to do creative visual effects, because that's where the real fun stuff happens. But to be honest, because, you know, VR productions are usually budgetarily challenged. Um, sometimes you don't have, you know, the coin to get even have fun. Um, you're just, you know, spending it all on taking out lights.

Paul DeNigris: 32:10

Yeah. So, so VFX is. Part and parcel of the whole process, right? It really is. Okay. So, so therefore VR creates, you know, typically what I ask a, a guest is what are some challenges and problems that VFX helped you solve? You, you just covered that, right? Yeah. I mean, effects is, is helping you guys all the time just to, just to make VR.

Matt Celia: 32:33

If you want to make VR before you go make anything in VR, the key thing is you should take like a beginner intro visual effects class. Just to understand the basics of how stuff works. You should understand, like, what a clean plate is. You should understand how green screen actually works. You should understand how compositing works. Like, you don't need to be an expert artist and any of that stuff. Like, lord knows I am not, like, the world's best roto artist, or composite keyer, or whatever. But I do understand the basics, so that when I'm on set, I know, like, hey, I need clean pixels there. And if I don't get those clean pixels, it's going to be a really expensive cleanup job. Because if you think about it, you have 180 degrees field of view. If you want cinematic lighting, you're going to need to put lights in your frame. Like, or you're going to need to spend a lot of money on very large sources, but lighting with super large sources far away from the talent means that you're not going to get the look you want. So you'll always paint out lights. Well, now you want to move the camera. Okay. Well, that just became a lot harder because how do you capture a clean plate and move the camera and do that paint? That becomes really challenging. Oh, you want your light over here? Well, guess what? Your whole camera shadow is now on this opposite wall. How are you going to clean that up? And so, it is a frustrating, like, like, jigsaw puzzle. Like, you just, you know what it is? It's kind of like you're in the, you know, submarine and a pipe bursts and you put your hand on it and then another thing bursts and you gotta go put your hand on that and then like, and like, that is All these things in visual effects is the answer to solve all of these problems. But if you, um, if you don't shoot it correctly or think about it, well, you know, every hand that you're covering those pipe is dollars and your visual effects budget can quickly spiral out of control.

Paul DeNigris: 34:28

For sure. And then on the flip side, having to shoot for VFX creates a whole other set of problems, right? Because like you said, you have to shoot clean plates. You have to shoot green screens a particular way. Uh, and sometimes it's playing whack a mole. It's this, this, the same, the same metaphor. I remember like, uh, we had those drone shots in the, in the hedge maze. And we go back and forth and I'd fix one thing and you'd be like, here, this is what I want you to fix. And I'd fix that in the process of fixing that I've knocked this other part of the shot completely out of whack. And you're like, well, that was good, like three versions ago. Now it's all fucked up and,

Matt Celia: 35:07

and, and, and it's hard because as you're doing it, you just version, version, version. I mean, we go through stuff so often and I will say this too, in VR, because there's such a huge frame of stuff. Just QCing your own work is challenging because you'll be looking at this one part of the frame and be like, yeah, that's great. That's awesome. And you'll watch it four or five times and you'll totally forget about this part of the frame. So, you know, if I were to distill this conversation down to one sound bite of advice is just give yourself more time, more time to capture on set to get the assets, more time in post to QC it, to watch it because every part of the process has friction and there's just so much more that you're creating. And you're working, you're working at 8K resolution. It's huge. You're working at 60 frames a second. That's almost three X a normal film. That's that's a lot of data. And so, and then your frame is enormous. So it just takes time. It's not that it's so much harder as much as it is. It just takes more time.

Paul DeNigris: 36:16

Yeah, it takes more time to send the files to a, to a vendor, to, you know, to download even within your internal team, you know, offsite having to download stuff for, for,

Matt Celia: 36:26

I mean, that's primarily why at LightSail, we centralized everything.

Paul DeNigris: 36:30

Yeah.

Matt Celia: 36:30

And we have people dial into our computers because we were spending days just copying files to a hard drive and then shipping it off. And then, oh, wait, there was a problem with that. Oh, well, I got to do that again. You know, their drive and like. You start to lose a lot of time.

Paul DeNigris: 36:45

Yeah, I can imagine. So you talked about, uh, John Ross, the director of a faceless lady and his uh, facility with moving the camera planning camera moves. Can you talk about, you know, like what are, what are some of the, the camera rigs that you guys used? Are we, are we, are we steady cam? Are we Dolly? Are we on cranes? Are we on drones? You know, how are we getting these cameras?

Matt Celia: 37:07

So, One of the big things that you have to think about with VR is you need it to be as stable as possible. Um, if you're, if you're having multiple axes of rotation, it's a good way to make your audience sick, right? You want to let the, the audience be in control of pan, tilt, and roll. You want to control things like forward dolly. Lateral dolly, you know, you want to control some of the bigger elements, but you don't want your camera just doing like a little bit of this because they're going to feel like their whole world like doing this. So I will say that, you know, we stabilized over 100 shots in The Faceless Lady. Using Syntheyes. And it was a large portion of the work that we did, because even when you're on a Ronin, like a good old Ronin rig that has like the, the skeleton arms and is like a really good operator, um, there's still like a little bit of that shake and a little bit of that vibration and a little bit of the forward and backward and six axis movements. And that can be really, really disturbing in VR because you kind of feel like you're doing this. Even though your body is not actually moving and that separation is what creates simulator sickness That's the one word for it, but really it's your like your nervous system saying like something's not right here So I should feel sick So when we think about moving in camera, we typically stay on dollies because they're rock solid dollies of course are really challenging because You'll see the track right underneath you. So now again, you got to paint out that track. So what we'll do is we'll, we'll put the dolly on a long jib arm and that'll be kind of what we rock. So that way we just need more space, but we at least can pull backwards and not see the edge of the track. Um, and that's a really effective way to move the camera around, to be honest. Uh, cause we are able to get these like nice and subtle moves. We're able to control the speed, we're able to control acceleration and deceleration. So it's very, very soft, very, very linear as much as we can. And a lot of that is why we have headset preview on set. You know, John is in the headset, I'm in the headset, while we're capturing this, and this is why we did so many takes. Is like, we just had to get it just right. And we had an amazing crew in Ireland. Like we had just an incredible grip team, incredible camera operators. Um, but it, it is unforgiving to move the camera. Um, and John moved it a lot, which was great, but we wanted to make sure that every movement is so motivated about being in that right spot because. You have to remember, even when your camera's static, it'll still feel like it's moving because your audience is looking around. So they're doing their pans, they're doing their tilts.

Paul DeNigris: 40:07

Yeah, it's really, the movement's really effective, especially in the, in the hedge maze sequence, you know, because, because you're in, you're in that space and you've got the hedge walls on either side of you and the cameras pushing down the, the, the, the row or, or pulling back and it's, and it's going by, did you get, were those dolly shots as well? Or was that...

Matt Celia: 40:26

so the hedge maze was not dolly shots. The hedge maze was on a Ronin with like these like skeletal arms and a Rua running through the maze. You know, trying to keep it as steady as possible. We did a lot of post stabilization on that stuff to clean it up and make it look as comfortable as possible. There was moments in that episode though, where it's definitely not comfortable. And that's the conversation that John and I would go back and forth with a lot. We'd say like, Hey, you know, we can only use this much of it in the cut. And then we need to sandwich it between two comfortable shots. Cause we like to approach VR as like having a comfort budget. Whereas you can have one uncomfortable shot, but if you have like three uncomfortable shots in a row, guess what? Your user is just, they're turning off the headset. They're getting out of it. And you don't want that. You want to be able to use it to say like, I'm being chased by a ghost. I should feel a little uncomfortable. Not like I feel queasy and nauseous and need to go lie down for an hour. Yeah. That's a great point. I mean, movement is one of those things that I think. People are very, in VR, they're very scared to try. And they're also told that they can't do it. And the reality is, the truth is it's somewhere in between. And I think John Ross figured that out. I think he found that, like, threading of the needle, where he created the sense of movement. And you know, interestingly enough, he actually looked at Citizen Kane as a touchstone. Um, because when you look at that movie, Those cameras were enormous. You're not like popping those on a little like gimbal and flying around places. They're huge cameras. So all your movement has to be careful and controlled and orchestrated like almost a dance. And then also that film surprisingly has a tremendously deep depth of field. Like you'll see all the way out the window and he's like, you know, On his sled as they're, as they're talking in the foreground and they're moving the camera and the characters are moving towards it. And him and I looked at that film a lot and we like broke down certain sequences about like, this is how the staging should feel. And that became a real touchstone for The Faceless Lady that I think worked.

Paul DeNigris: 42:39

Well, there you have it. All for all of you. Citizen Kane haters out there who, who complained that it's constantly the number one greatest movie of all time on everybody's lists. It's pretty good. It's a pretty damn good textbook, even even now, even in VR in technology that that Orson Welles. Couldn't possibly have dreamt of.

Matt Celia: 42:59

But has a lot of the same challenges. Like VR is about taking what's old and making it new again, you know? Um, because it is the birth of a new medium. It's not going to replace TV movies or anything like that, but I think it's fundamentally going to be a different way for audiences to experience stories. And that's what I think is really exciting about it is that we start to walk this line a little bit between storytelling and story living.

Paul DeNigris: 43:23

And that gets really exciting being that our target audience for this podcast and our established listeners and viewers are indie filmmakers. They don't have the budgets of Meta / Crypt TV / Eli Roth, but they want to make something in VR. They've got a story that that screams out for VR. And you said, step number one, take some basic VFX training, learn, learn that. Okay, what's the next step? What, how, how does, how does somebody jump into this?

Matt Celia: 43:57

Yeah, it's a great question. I mean, the first question you have to ask yourself if you're creating something in VR is, is my story better in VR? And that sounds like a, a dumb thing to say, but you'd be shocked. Um, you have to have a good reason to want to put people there. Like, I think Robert Zemeckis has a new film coming out, I think called Home, where it's like a single shot in a house through ages of time. And I look at that and I go, that should have been in VR. Because, like That just feels like so right for the medium. Um, but a lot of folks come into our door and they say, I want to make Hardcore Henry. I'm like, Hardcore Henry is not in VR. Like that's a first POV. It's all about like moving the camera and crazy stuff. The, the key to start out is to get a camera and start looking at content, you know, getting into the ecosystem of VR does not have to be tremendously expensive. Um, a really good, still pro level quality can be found with like the Canon EOS R7 with uh, the 3. 5 millimeter dual fisheye lens they have. It doesn't do full 180 degrees, does like 150 degrees. Um, it's not a full frame sensor, it's a crop sensor. It's not 8K, it's 4K. But, that means that the files are a lot easier for you to work with. Like, because working in 4K is tremendously easier. Working at 30 frames a second is tremendously easier. And because it's not a full 180 degrees, you don't really have to worry about the visual effects work of cleaning up your tripod, or taking out the other lenses on either side, or all that stuff. And you can just start figuring out, like, where to put the camera. Because that's the key. Where do I put the camera? How do I move the camera? How do I block to this camera? How do I tell stories with this camera? You need to understand that before you do anything on a huge scale, for sure. And you can make really quality content with a system like that. And that system can be had for like under three grand. And, or you can rent it on lens rentals for like a few hundred bucks. And, um, get into that and just start messing around with that. I think Canon's tool to stitch this stuff, uh, is like five bucks a month. integrates directly with premiere. Um, so there's, you know, there's definitely, there's definitely levels to it, but getting in and starting to develop your own visual language is step number one.

Paul DeNigris: 46:26

Uh, what's some VR content that you're consuming right now? That's got you excited. What are you watching? What are you playing?

Matt Celia: 46:34

Uh, great question. Um, I spent a lot of time in headset looking at the stuff that we make at LightSail, so I feel like sometimes I don't get out to see what, what else is out there. Um, but two recent things that I really enjoyed is, um, on Apple vision pro there's an experience called What If. Uh, which was created by ILM and, uh, and Marvel. And it is a great example of story living because, you know, they come into your living room and they take you into these worlds and you have some light interactivity. Uh, and it made me think. feel like I wanted to get back into the Marvel universe and I wanted to, like, kind of be a superhero again. And like, like that was very, very cool. And I thought that the team there did a really good job. They just won an Emmy for it. Um, I obviously have been watching a lot of our Live in Red Rocks series, uh, which is concerts. Cause I think music works so well in VR because you're, you're on stage in a vantage point. You can't even buy a ticket to. And you get to see incredible performers making incredible music. It sounds awesome. You're at Red Rocks. You're in this iconic venue. So I really love that. Um, and then let's see, what else have I been doing recently? Uh, I play a lot of mini golf. Walkabout mini golf is one of my favorite things and easily the thing I turn most people on to when they're getting started in VR. Um, and oh, so this week is Venice, uh, the Venice Film Festival and they have an amazing immersive, like, island. And so last week I had the opportunity to, uh, do a dress rehearsal, uh, from a group called the Ferryman Collective and they have a new immersive theater piece that's done in VRChat. And that was incredible because I love theater and, uh, they've managed to create these amazingly beautiful worlds that you get thrown into as an avatar and get to interact with and feel like you're part of the story. Um, and it really takes that idea of live story living to the next level. And few things are, I think, as groundbreaking as the work that they are doing, uh, in VR. I think that they're just telling some incredible stories. Um, and it's really fun.

Paul DeNigris: 48:47

That's great. Uh, what's up, what's up next for you and LightSail? What, uh, projects do you guys have coming up?

Matt Celia: 48:54

Um, so here at LightSail, you know, we're doing our Live at Red Rocks concert series, uh, and we've been busy filming season two of that, which is very exciting. So there's a lot of like amazing acts with that. We've been doing, um, some workflows with Apple Vision Pro, which we're pretty excited about. It's all HDR. So that's a whole nother wrinkle to things. Um, uh, but you know, pushing that quality level forwards, pushing the resolution, pushing the color correction. Um, and also building tools for a lot of this stuff, because as we encounter these problems in production, like how to live preview or how to like even review visual effects cuts, we find that there isn't a... a defined ecosystem of tools to help reduce the friction for artists. And so we've been heads down on a couple of tools that I, you know, I hope are going to keep making things a little easier so that we can stay being creative and less technical.

Paul DeNigris: 49:50

Yeah. You guys have that app, uh, for the, the quest. Screening room integrates with frame.io.

Matt Celia: 49:58

Yeah, exactly. So we're working, we're working on revamping that for, um, adding a couple of much requested features for that. Um, because it's just, we found it so helpful. It's probably the app I'm in most on my headset, to be honest, because I'm constantly reviewing stuff and like making notes and watching down cuts. And, uh, And it's really, really fast and easy to do that.

Paul DeNigris: 50:21

Yeah. I remember, uh, you know, some, you would review our shots through that system and you must have a macro or something, uh, or something that you can key quickly when you're in the headset that just said VFX issue. And then, and then you'd go back and put the note, but I'd see them in real time. And it'd be like, that's the VFX issue. And I'm like, What's the issue? I know.

Matt Celia: 50:44

Yeah, so we have, we have like hotkeys, um, just to do things quickly when I don't have a keyboard, but the app actually has like keyboard support. Oh, nice. So, um, when that's all like hooked up and doing that, I actually just write the notes in real time. Oh, that's great. Um, which is, which is awesome. But I love to add like annotation support so I could circle things and, you know, anything that helps artists like understand the issue quicker is good because we just want to, you You know, VR is challenging enough as it is, so anything we can do to kind of make it easier, I think, uh, helps make projects look better.

Paul DeNigris: 51:16

Yeah, that's fantastic. Uh, anything else you wanna, you wanna plug? Um, you guys got an Emmy nomination?

Matt Celia: 51:24

Yeah, um, we're, we're, we're stoked that our, our show Live in Red Rocks VR is nominated for a Primetime Emmy. Um, so I don't know when this airs, but, um, hopefully we'll, we'll have one, but it's just an honor to be nominated. There's so many amazing, um, creators in the category this year. Uh, and I just think that it's a good time for immersive video. And I think that people are excited about it. And I think that there's. A lot of stories that work well at it. Um, and we're just excited to keep pushing that immersive ball forward at LightSail. So it's a great, it's great fun.

Paul DeNigris: 51:57

Yeah, that's awesome. You guys have had a really, a really good couple of years between the Emmy nom and, uh, just, just the great reviews on The Faceless Lady. You know, calling out the, the excellent, excellent technical work and the storytelling, uh, work. So yeah, you guys are, are doing great stuff and, uh, um, tell us, tell the viewers, you know, where they can follow your, follow you on social media or find your stuff on the web.

Matt Celia: 52:22

Yeah. So, um, you know, always check out our website, lightsailvr.Com. And see most of our latest work. If you want to follow me personally, I'm on Instagram. Most days as a immersive_Matt.

Paul DeNigris: 52:34

All right. Well, thanks so much, Matt, for sharing your, uh, your wealth of experience in, uh, virtual reality and filmmaking and VFX. It's like the whole package. Uh, I'm sure that the, uh, my audience appreciates it. And, uh, and I appreciate your time.

Matt Celia: 52:50

Cool. It was fun. Thanks for having me on. It's always good to geek out about VR and storytelling and, you know, let's do it again.

Paul DeNigris: 52:56

Absolutely. Thanks for Foxtrot X-Ray and the VFX for Indies podcast. I'm Paul DeNigris, thanks so much for watching. Remember to like, and subscribe and check out our back catalog at vfxforindies.com. See you next time.