Q&A with Academy Award-winning VFX supervisor Joe Letteri

Inside The Film Room senior writer Johnny Sobczak spoke with visual effects supervisor Joe Letteri at Film Fest 919 in Chapel Hill, N.C. on October 13, 2019. Letteri is a four-time Academy Award-winner and a pioneer in the field of visual effects. His filmography includes “The Lord of the Rings: The Two Towers,” “The Lord of the Rings: The Return of the King,” “King Kong,” “Avatar,” and the “Planet of the Apes” trilogy. He is the director of Weta Digital in New Zealand, one of the world’s premier visual effects companies.

Johnny Sobczak: What are your day-to-day tasks at Weta, and how hands-on are you with the films?

Joe Letteri: Supervising is more of an oversight role. We treat it very much like a film. We start in the morning looking at dailies. We figure out what we like and don’t like and then figure out what we need to get done for that day, for that week, for that month. It could be artistic or creative, what we’re trying to get into the film, or it could be a lot of technical, research and development things that we haven’t figured out yet, that we still need to advance. 

So, there are discussions that happen at multiple levels, but it is still generally like you’re shooting film. In the old days, you would send things to the lab at night, come in the morning, see what you’ve got and if you need to reshoot anything. That was called dailies. We kind of do that every morning, we work all day, but instead of going to the lab, we send things to our render farm, which is a bunch of computers that generate all the pictures. So, we come in in the morning, look at them, figure out what we need to do. But we keep that same rhythm like traditional filmmaking.

JS: You’ve been working in this field forever. You worked as a graphics artist on the first “Jurassic Park,” which was one of the films that really started integrating CGI into live-action. What exactly did you do on “Jurassic Park,” and when you were working on that, did you foresee the trajectory of this field? “Avatar” came out just 16 years after “Jurassic Park” and was almost completely computer generated.

Letteri: At the time, we were just trying to figure out how to make it work. I was just interested in using computers to make pictures, learning what makes things look natural, what makes things look photographic. When “Jurassic Park” came along, that’s what I focused on. How do the dinosaurs look? What’s the quality of the surface of their skin? What do the eyes look like? What do the claws look like? How does something look real? That immediately ties you into the cinematography aspect of it. You have to figure out how to make the lighting match. Light has to hit a surface and reflect off of it. If that surface is painted or if it’s wet or if it’s skin, it looks different. Why? Suddenly there’s a lot of research to learn all that stuff. 

Overall, I created the framework for how everything was going to look. I did a lot of the lighting myself, like the T-Rex coming through the fence. I lit that whole sequence. The first Brachiosaurus sequence? Those couple of shots were my shots. It’s a combination of hands-on and R & D, and it kind of always is. But these days, I do less and less hands-on because the team is so big.

JS: It’s interesting you were talking so much about the light and how it’s hitting and interacting with the characters. Typically, in a film where there aren’t so many CG elements, the director of photography is the one concerned with how things are lit. How much do you all collaborate with the DP specifically in knowing exactly how the light needs to look?

Letteri: We are doing that more and more these days because the tools have become so much more realistic. In years past, it was much more difficult to do that because the tools were so much more specific to the technology. 

JS: I feel like I would be worried about stepping on toes if someone wants a shot to look a certain way and what you’re doing at Weta is not connecting.

Letteri: Generally, DPs are pretty happy with what we do because, first off, if a lot of shots are starting with live action, we’re fitting into the live action. We’re adding where we need to because they don’t have the character in the scene when they shot the background. So, we might add in eye lights and hair lights and things like that, that you know they would do anyway, and you can just figure out the style from either talking to them or the rest of the footage or other work that they’ve done. Sometimes it’s the directors themselves that set the lighting style, so it really depends on the project.

JS: With “The Lord of the Rings,” I know you came in and you worked on “The Two Tower” and “The Return of the King” and won two Academy Awards. When they actually made those films, Peter Jackson filmed all three films consecutively and simultaneously, so how was it coming in after they had already finished “The Fellowship of the Ring,” which you weren’t involved with?

Letteri: It wasn’t really a problem because the team working on it knew what they were doing when they shot it and had collected all the information we needed, and obviously Peter was there to talk about what he wanted. My focus was really Gollum, and that was the big change because you saw Gollum in “Fellowship,” but you only got a glimpse of his eyes and his fingers. He was designed kind of like a monster. He was designed to be scary. But when we got him and started to need him to perform, we realized we had to alter the design because he couldn’t perform because of the way he was looking. Fortunately, you didn’t see enough of him that we were locked into the design of the first film, so that’s really what I spent most of the first six months doing, trying to figure out how to change his design. You almost do like screen tests. Change the design, try some different actions and performances and see if it works, evaluate the design and keep changing it until you have a design that will do what you need. 

JS: You talked a lot about developing Gollum and how much work went into that, especially with the sub-surface scattering technique that you created. Thinking about how vital that is for bringing these characters to life now – especially Thanos in “Avengers” and Caesar in “Planet of the Apes” – how has that technique been altered since “The Lord of the Rings”?

Letteri: Oh, it’s changed a lot. We used a very approximate technique to do it the first time. That’s all we understood and technically could do, given the computing power and the software at the time. We’ve all since developed it so that it is fairly accurate to what happens in the physical world. If a photon of light hits the skin, it bounces around off of things inside the skin before it comes out again. We trace all of that, so there are millions or billions of computations that go into each frame now. 

JS: You came to Weta and did “The Lord of the Rings” right away, and then later “Avatar.” A lot of people praise “The Lord of the Rings” for its use of miniatures and practical effects, as well as how you managed to integrate those effects with the digital. 

How do you approach a project like that, with real-world implements that are sewn together with CGI, compared to approaching a project like “Avatar,” which is almost completely CG? Is there a difference with regards to pressure?

Letteri: It’s really more about the requirements. The more we learned about making things look real, the less we felt the need to do a miniature. Miniatures have their own limitations, as well. It’s hard to light them, it’s hard to do camera moves. It’s also impossible to change them once you’ve shot them. For example, we worked with miniatures all throughout “The Lord of the Rings.” Barad-dûr (Sauron’s tower) was a huge, incredible miniature. Even though it was meant to be like a mile high, it was still six or eight feet tall. But in the end, we had to destroy it, and you couldn’t destroy that miniature at scale. It was too small to get the believable scale, so we had to create a full, digital recreation of it to destroy, which meant we could have used a fully digital one the whole time. We just didn’t know it when we started.

JS: Speaking of miniatures, I love the workshop branch of Weta, especially with the work they did on “Blade Runner 2049.” What’s the relationship with the work they’re doing there, and how much collaboration is there? 

Letteri: It depends on the film because we didn’t work on “2049,” but obviously on “Rings” there was a lot. We don’t do as much miniature shooting these days. Most of the collaboration is with regards to character design or costumes and weapons. 

JS: The resume of directors you’ve worked with is amazing: Steven Spielberg, Peter Jackson, James Cameron, Matt Reeves. How collaborative is that process, and how might it vary from one director to another?

Letteri: Generally, it’s pretty collaborative. Good directors like those have two things in common: they absolutely know what they want – which you would think leaves no room for collaboration – and they want it to be as best as possible. So, they’re open to saying, “Hey, if this will make it better, I’m willing to listen.” You kind of have to do both. You have this sort of minimum level you have to hit where you know you’ve got the story, you know the director will get through it with this moment because that’s what he or she asked for, but then if you can come up with anything to make it better, directors generally don’t say no to that. 

JS: What is your favorite experience from all the work you’ve done or the film you’re most proud of?

Letteri: That’s hard to say. I don’t think I could nail it down because they’ve all brought great things to them. “Jurassic Park” was an eye-opening experience, or even the Star Trek I worked on (“Star Trek VI: The Undiscovered Country”) was great. I really can’t complain because each film has allowed me to do something new. “The Lord of the Rings” was great in that way, or even the first “Mission: Impossible,” trying to figure out the realism of doing the helicopter crash and building that tunnel. “Avatar” opened everything up to a whole different world, “Tintin” and “The BFG” were just really appealing with nice characters in a different way. Every film just brings its own unique aspect to it.

JS: You’re the industry standard for this work. A lot of people look to you for inspiration and to see where the bar is set. Where do you personally look for inspiration when you’re going through a process and thinking “Well, can we make this work?”

Letteri: A lot of it depends on the story. If the story works, we can make the effects work because that gives us a big clue as to where we’re going and where to put the effort. Otherwise, we look at a lot of just real, natural phenomena. We’re constantly studying things in the real world, in both large-scale detail and small-scale detail, to figure out how to pull it all together to make it look real.

JS: One thing people talk about a lot is how effects age. Some age poorly, some age well. Is that something that you consciously think about while in the process of creating?

Letteri: You do, but that’s the target of the realism. I remember that mostly on “Jurassic Park,” because we didn’t really know what we were doing! So, we were just trying to figure out what makes it look real and just hoping it looks good in five years.

JS: Are there any filmmakers right now that you’re a fan of and you’d like to work with?

Letteri: I think I’m going to be doing “Avatar” for the next few years. So, I think I’ll be working with Jim (Cameron) to finish those off. 

JS: The first “Avatar” came out ten years ago. What are you noticing now with the sequels that is most different from the original?

Letteri: The only thing I will say is that the stories are a lot bigger than I thought they’d be. Jim does not disappoint when it comes to story, so I think there’s a lot to look forward to. We have two more years before the first sequel comes out.