Interview with Yoyo Munk

Transcript

0:00:00 - 0:01:08
Paula Bray
Thank you. Yoyo Munk, for joining us. We have Yoyo, is a scientist - artist. We have Paul Duldig, the CEO of the State Library Victoria, and I'm Paula Bray, Chief Digital Officer here at the library. State Library Victoria acknowledges the traditional lands of all the Victorian Aboriginal clans and their cultural practices and knowledge systems. We recognise that our collections hold traditional cultural knowledge belonging to Indigenous communities in Victoria and around the country.We support communities to protect the integrity of this information gathered from their ancestors in the colonial period. We pay our respects to Elders, past and present, who have handed down these systems of practice to each new generation for millennia. Thank you for joining us at the library today for our In Conversation series. This series we have visiting technologists, futurists, artists, researchers and library staff in conversation to explore how new, existing, and emerging technologies are shaping the library of the future.So thank you so much for joining us.0:01:08 - 0:01:10
Yoyo Munk
Thank you so much for having me.0:01:10 - 0:01:16
Paula Bray
Yoyo, can you just give us a little, background into your, your practice?0:01:16 - 0:03:13
Yoyo Munk
Of course. So my background is, as a scientist, I, studied as a, physicist originally, and then, transitioned into working in biology for my graduate work.I focussed mostly on insect flight, aerodynamics and, navigational behaviours and neuroscience for my academic work. And then transitioned into, working with augmented reality after leaving my postdoctoral work. And for a while I spent, I spent several years at the start-up Magic Leap in the United States, where I founded and led the Quantified Experience group there, which was doing experiments on human vision and visual responses to, augmented reality displays.And then for the last six years, I've been working to create content for mixed reality exhibitions. Primarily in collaboration with the content collective Tin Drum. We produced a number of pieces that have been publicly exhibited. The first piece that was shown, as the The Life, which was a volumetric piece, featuring the performance artist Marina Abramović, which, showed at the Serpentine Gallery in London in 2019. In 2021we showed, a piece – where I was the primary artist and creator of it called Medusa. And that showed us the headlining show of the London Design Festival and featured in the Victoria & Albert Gallery in London. And then most recently in 2023, Tin Drum produced, a, a very ambitious, volumetric piece called Kagami, which has toured the world and most recently showed as part of the Asiatopa festivalmost recently here in Melbourne.0:03:13 - 0:03:30
Paula Bray
Yeah, it was fantastic. Both Paul and I have, experienced Kagami and, really enjoyed it. Can you tell us a little bit about working in mixed reality, as opposed to VR, when creating works like you did for Kagami and Medusa?0:03:30 - 0:06:50
Yoyo Munk
Sure. Okay, so, mixed reality. I mean, so I mean, mixed reality means different things to a lot of folks.And so I guess when I talk about mixed reality, I'm usually thinking about a headset technology that features an an optically transparent display. So when you wear one of these devices, your vision of the real world is unobstructed. At least in the sense that the the light that is being reflected off of objects in the real world is still going into your eye without going through a camera in the meantime.So, this is distinct from camera passthrough style devices, where it's a VR helmet featuring a live feed of a camera from the outside. And the devices themselves, are capable of, showing virtual geometry, overlaid, with the physical world which you are inhabiting. And I think, like, so, you know, mixed reality in that sense, like the idea that you are, you know, taking three dimensional virtual geometry and trying to integrate it into a real world space is vastly more difficult than doing VR.So, like one of the benefits of VR is that, like, you essentially have complete control over the sensory apparatus of the, human that's involved, right? Like you are replacing the visual inputs to the eyes. If you have a nice big pair of headphones, like, you know, you're controlling the sound. You know. Yeah. ????? of sensory and stuff.Let's leave that aside. In the mixed reality sense, you're having to make a piece that that where you've got virtual geometry that is trying to integrate into that real world space. And there are immense challenges that come with trying to make that geometry fit in a way that feels cohesive and it's not jarring. And, and creating sort of like a uncomfortable sort of like sense of friction between the virtual geometry and the real worlds that people still very much feel a part of.There are some big advantages of it, I think. And one of the most important things from my perspective is the, the opportunity for the sociality of the experiences that you create through this method. It is certainly possible to, have, you know, VR experiences where there are many people wearing VR helmets in a room and you're kind of have some kind of avatar representation of other people and where they are in that space.But like, this is a very this is a tiny, microcosm of like of all of the information that is encoded with the sort of like non-verbal communication between humans and crowds. And the idea that, you know, you are you still have this sense that you are inhabiting an exhibition space and have that sense of nonverbal, cohesion, with the audience that you are a participant in.I think is really it's it's it's hard to I think it's hard to overestimate in terms of its importance and to creating a, an experience that that feels impactful.0:06:51 - 0:06:52
Paul Duldig
Can we, can we talk then about Kagami?0:06:53 - 0:06:53
Yoyo Munk
Yeah.0:06:53 - 0:07:10
Paul Duldig
So both Paula and I saw Kagami, and it was a really interesting experience. And that I think what you're saying certainly came through Kagami, where as a mixed reality experience, it wasn't really using much of the room.It could have been anywhere.0:07:10 - 0:07:11
Yoyo Munk
Yeah,0:07:11 - 0:07:29
Paul Duldig
but the the fact that you could see other people experiencing the same thing you're experiencing and walking around. Ryuichi, and, you know, sitting down and just looking and and so on. So it was more of a social version of mixed reality, perhaps.0:07:29 - 0:09:04
Yoyo Munk
Yeah. I think like the what I like most about, an experience like, Kagami is – I mean, I think if you're, if you're looking for, like the kind of detail that you might be able to find in sort of like, you know, an exquisitely filmed performance or like the the kind of sound that you would get from a studio recording.And it's like a piano in that room. There's, there are like, this is not it's going to, it is not intended to surpass or replace any of these forms of communication. Like there are many, many valid ways of interacting with the work of any particular artist. But where I feel like this can be most successful is when you are in a room with a small number of other people, and you and some of the artifice kind of melts away at some point, and you feel that you are in a room with Ryuichi playing a piano, for a small audience of people.Like, I love standing around on the outside and just seeing the people who get up out of their chairs, and they go and they just sit down cross-legged on the floor because, like, you know, they get to sit right next to the piano bench and, you know, there's, you know, it's in the one sense it's not real. I mean, it's, you know, the, the image there is virtual, but it is I mean, the feeling of being there is difficult to achieve through other methodologies.0:09:04 - 0:09:11
Paul Duldig
It might be with just, just give us a quick, reminder of what Kagami is and then perhaps do the same for Medusa...0:09:11 - 0:11:14
Yoyo Munk
So Kagami is a, is a performance a piano performance by Ryuichi Sakamoto. It was volumetrically captured in a studio in Japan. In 2020...late. And in. It's the way the experience unfolds is you have a, a circular arrangement of chairs. An audience is invited into that space. And so all of the participants in this experience start out, seeing, Ryuichi at a piano playing in the centre of the room. And it proceeds linearly through Aves, a selected curation of songs like it has.It's it's very much a show. There's a performer, it has, you know, a sequence that runs through and is like, you know, it goes through the same way each time in terms of how the songs play out. But the feel of any given performance, is very different in terms of how the, the audience of that day interacts with that space and how they choose to inhabit that room, and interact with one another as, or like, you know, have that sort of like ephemeral connection between strangers that you have in live performance.And so, you know, it, I think it's something that, feels like it has a more inherent sense of individual agency within that audience. That is somewhere in between the idea of, like, being at a live concert, like having that, that sense of energy of a performer in the room that, like, you know, that you feel like you are there, you are seeing something that's happening right now, versus like the idea of sort of sitting by yourself and listening to music on headphones or watching a film, in the comfort of your own home.Like, you know, this operates within sort of a, a space in between.0:11:14 - 0:11:16
Paul Duldig
It could have been VR as well, couldn't it?0:11:16 - 0:11:50
Yoyo Munk
I think that, like, I mean, certainly from a technical perspective, there's nothing that would prevent this from existing inside of a VR space. I think one of the things that you would lose in that, case is, oftentimes, I mean, so certainly, like, you know, for the ability to do a large audience, like, we would have people bumping into one another quite a bit more, inside of that room.So like the, you know, the fact that these devices are transparent, is helpful in terms of being able to just, like, ensure some degree of audience safety.0:11:50 - 0:12:07
Paula Bray
I really enjoyed that fact that you could you could see people in layers in, in between the visuals and, and watching the hands of the piano play. Yeah. She found that quite, a good part of the experience.0:12:07 - 0:14:47
Yoyo Munk
It's one of the I mean, it's it's certainly something that that I... I like that aspect of this piece quite a bit. I mean, you had asked a little bit about sort of like Kagami and Medusa, and like in... Medusa, I mean, so in Kagami, we sort of, like, operate in a fairly dark room.And it's like, as you say, like, you know, the actual environment of the room itself is background. It's not something that is intended to like, you know, there is a mixed reality component to the idea that you see your fellow audience members. But like the room itself, aside from, the theatrical lighting, that kind of like backs up some of the, the virtual effects that happens in that space.The focus visually is really on that virtual aspect, on Ryuichi actually playing in the centre of the room. For Medusa, this is a fairly different kind of idea. So this is not a volumetric captured piece. This is more of a generative, evolving experiment within, of the idea of trying to do architects and architecture and structure, within a virtual space.So, like taking the idea of a environment that has a sense of volume and a sense of of occupancy within a room, but completely removing any physical aspect to it, like this is a purely virtual, sculpture slash piece of architecture floating in that room. And there the idea was very much to, like, we like the lights are bright in the room there.And the, the room itself that contains the sculpture is a very important and integral part of how the piece works. So it's a very minimalist, piece. It is composed of, like, you know, just shy of 10,000 hanging columns that move through the room and, subdivide space, and, change the way that they, form themselves into a, emergent structure, based on how the structure is inhabited by the audience in real time.And but at all points, like, you know, the, the focus, like, you know, as your vision is drawn up through that structure, like there is the palpable background of the environment in which it is housed. And that's very and it's designed to be so that the architecture of the room that it inhabits is an equal player.In the total architectural experience of the piece.0:14:47 - 0:14:49
Paul Duldig
Yeah. Interesting.0:14:49 - 0:15:53
Paula Bray
One of the things for me at Kagami, it was the introduction. You walk, you know, you have an onboarding experience before you go into the the room, which is quite beautiful. It's got, scent, it's got amazing visuals. And you stay in there for a little bit of time, but then when I was experiencing, the next phase in the room, Todd actually came in and, and, and talked a little bit about the experience for the audience and described it as, not perfect.You know, it's, you know, it's you need to forgive a little bit for some of the experience in terms of the, you know, you're working at the cusp of advanced technologies. Can you talk a little bit through how that sort of onboarding with using advanced technologies in, in those experiences, whether that's important or that visitors appreciate that sort of an in introduction?0:15:54 - 0:15:54
Yoyo Munk
Yeah.0:15:54 - 0:15:54
Paula Bray
No?0:15:54 - 0:20:19
Yoyo Munk
Yeah. I think that, you know, it is it is certainly like it can be jarring just to like throw a headset on, on someone and just be like, all right, off you go. Like the idea of preparing the ground and, like, and having a space where, like, you are, transitioning into, like, whatever experience is about to happen.I mean, this is not something that's unique to headset based experiences, right? Like, this is, like that idea of, like, like having some degree of expectation setting of, like, having this sort of sense of welcoming, especially when you know that what you're asking people to do next is something that they're probably like, they don't really have a reference point for, like, this isn't like, you know, you're you're looking to essentially project this idea that, like, you know, what you're about to see is going to be different from things that you may have seen before.And just like, you know, it's going to, and but like, this isn't a threatening space. This isn't like nothing bad's going to happen here. It's all going to be fine. And, like, there's a it's good to just sort of like, have a little sort of sense of calm going into it. I mean, I think that like, with regards to, you know, speaking at the outset and talking about sort of perfection or lack thereof, I mean, I think it can go both ways.I mean, I think, like, you know, you want to avoid seeming like, you know, you're just like being self-deprecating for the sake of it. Right? And I mean, personally, I think that, some of it, especially within that space of volumetric capture, the imperfections for me, often lend a sense of poignancy that, that, enhances my experience of it, like, so...I mean, I guess for listeners who may not be like that, generally the idea of with any volumetric capture piece is that you are assembling a, geometry, a three dimensional geometry, that of of whatever happened inside of that capture space. And you're assembling that from the images that are taken from an array of cameras around that room.And, you know, this is an inherently algorithmic process. You know, there's nothing in this pipeline where, like, you know, something is sitting there saying, like, you know, it knows what a human is supposed to look like. And so it really all depends on what the camera's are able to see at any given point in time, and what their best guess is of being able to reconstruct the geometry of what they think was... what the thing was in the room at that time.And so you get glitches with it, like it's, it's not a, like there are, there is some noise that is inherent to this process, which is related to just like the technical limitations of what you can do with like eight cameras of a certain resolution with the processing time that's available. And so, but I think that there's actually something like really interesting to the about that, that glitching is to me because it has a character that reminds me more of film grain, than it does of sort of, of, for example, like a lack of skill on the part of, like an artist who might otherwise be sort of likehand sculpting these details. Right? You know, it's not something that that feels like, you know, someone did a bad job, in terms of, of, like, being able to, like, it's not like you're seeing a video game character and it's like, you know, oh, this is blocky or like at, like someone did this badly. But there's a noise to it that, like, reminds you of the artifice that's inherent to the capture.And I think especially when you're thinking about, volumetric capture of human performance, especially in the case where, like, there will be no further opportunities, to capture that individual performer. This... there's a sense that, like, you know, this is something that is of its time, like, you know, this represents the desire to try and capture something while the opportunity existed.And, you know, if you wait for things to be perfect, you'll never end up with anything. And so...0:20:19 - 0:20:37
Paul Duldig
And also when you think about from when I was looking at it, the, the difficulty, the glitch, perhaps between telling the difference between someone's fingers and the keys is actually quite a sort of beautiful metaphor for a pianist.0:20:37 - 0:20:37
Yoyo Munk
Right,0:20:37 - 0:20:43
Paul Duldig
anyway, right? Because they sort of embody they're at one with the, with the, with the instrument.0:20:43 - 0:22:14
Yoyo Munk
Yeah. And I think that like, I mean, that and that's obviously one of the most. So I mean, getting the hands right for Kagami, like, yeah, that was that was a lot of work. And, like, you know, ultimately you're trying to, like capturing hands in photogrammetryis already a reasonably difficult thing. Just because hands are complicated, there's a lot of geometry involved. It's really easy for one finger to get in the way of the other, and then suddenly you're not really sure what the shape of it's supposed to be. And, and I think that, like, like watching the hands move on the keyboard there and like, you know, those, those little kind of like, you know, it was in many cases it's it's pretty close.And in some cases you'll just sort of see like something drift a little further away than you would if it was actually a, human playing the piano at the time. And I think that's a, a reminder of just like the like this is an imperfect capture, but that doesn't reduce the significance of its potential meaning and its mattering.Like, you know, the ideal that photorealism or the idea, like, you know, of a simulacrum as being like a inherently required, component for, like, you know, for the attachment of, of human emotion on the part of a participant to an experience that they're having. Like, that's obviously false.0:22:14 - 0:22:27
Paul Duldig
It's almost like the memento mori and still life thatyeah, yeah, it's it's actually makes it more real. Yeah. Yeah. Beautiful. So can I ask, how does a scientist become an artist?0:22:27 - 0:25:21
Yoyo Munk
Through a convoluted, meandering path, filled with luck and happenstance and, you know, mostly just kind of falling into it backwards. I mean, I think that, a lot of the... Coming from science, I think that, especially, like working inside of biological spaces, where you're often dealing with data sets that, are inherently very high dimensional.There's a lot of noise inside of your measurements, and you spend a lot of time really thinking about how to take this noisy data set and extract a narrative from it. Right? Like you're like, it's very challenging to, to try and think of, like, hypotheses that are really easily tested, because, like the space of problems that exist or areas of interest within biological sciences that are really, truly amenable to being able to ask a simple yes / no hypothesis question is necessarily a subset of all the problem spaces that exist.And that doesn't make those other problem spaces any less compelling or, or interesting to explore. It just means that they're like less practical, sorry, tractable. From a and within a explicit hypothesis testing framework. And data visualisation is, you know, a very important tool in terms of being able to examine these, high dimensional data sets and try to like, look at things from a bunch of different ways and sort of try and see, like, you know, how we can find patterns within them.And this idea of like, of playing with around with signal and noise and, extracting, like emergent properties from, from data sets that are inherently complex. I mean, these still feed through into the kinds of things that interest me, within my artistic practice. Like, I like basically letting; I like, in my own work to, to have, to not have control over everything that is happening in the room.Like, there are ways that they're like the behaviours of the pieces that I'm presenting are inherently kind of a collaboration between, whatever ideas I put into it. And how an audience responds to it, within the space itself. And like that idea of, of having data sets, emerging from and like basically collecting data representing that data through visualisation, and trying to, like, distil those down into a narrative...there's not such a huge difference.0:25:21 - 0:25:23
Paul Duldig
I think that actually is an artistic process.0:25:23 - 0:25:23
Yoyo Munk
Yeah, exactly.0:25:23 - 0:25:25
Paul Duldig
And sculpture. Yeah.0:25:25 - 0:25:29
Paula Bray
So libraries have incredible datasets0:25:29 - 0:25:29
Yoyo Munk
They do.0:25:29 - 0:26:08
Paula Bray
Andyou know, as part of digital, it's quite it's a practice that you can really use those datasets to look at, you know, a bird's eye view of a collection. Or you can look at different lenses of a collection.And we've taken you on a tour of the library. State Library Victoria, this morning. We've had a little chat about collection and participation with audiences. Can you give us a little insight into how you could apply your science-art practice at a library?0:26:08 - 0:26:16
Yoyo Munk
Sure. I mean, I think that, like, I think it's a it's a big question.I mean, you know, it is a I would like to go0:26:16 - 0:26:20
Paula Bray
You've got, you know, you've got two minutes. No,0:26:20 - 0:30:23
Yoyo Munk
I mean, I think this and, obviously the State Library Victoria is among the most impressive examples of this in the world. But, like, you know, in any library of considerable size, like, I think that one, there's a sense of humility, that comes from, like, seeing this collection of immense amounts of human effort and, and immense amounts of knowledge painstakingly gathered over time, collected in a way, in, these rooms where, like, the sheer enormity of everything that has gone into not only like, like understanding the contentsof what's been, what's been kept here, but also the immense effort that goes into, distributing that and keeping it and making it accessible so that people can so that that information can continue to live on and that people can find it again. You know, it's, you know, there are you could spend many, many lifetimes, trying to work your way through, like trying to absorb the information that exists here.And this is just the stuff that we've kept, right? Like the I mean, like for all of the knowledge that is out there, like, when we think about what we have kept inside of a library, this is a, this is a living document of the collective attention and memory of how our culture has moved over time.Like the things that we have chosen to archive and the things that we have decided are important at any given point in time is this moving, evolving target. And so I think that like when I think about how... one of the things that I like to play most with and in terms of my own artistic practice is playing around with, spatial and temporal scales.I think that, you know, we are all human beings and, like, there are time scales and spatial scales that are comfortable in terms of, like what we can realistically deal with on an individual basis. And that and I think that one of the reasons why, like, libraries can feel like have that initial sense of overwhelm, is that like you are coming into an experiential contact with the, with a hierarchical level of human knowledge and behaviourthat is well beyond your individual capacity to absorb. Right? Like this is a collection of knowledge that is would be physically impossible for you to internalise and as an individual. And that's not the point of it. And so when I think about what can be done inside of a digital space, I think that I think about the fact that, for all that we know that libraries are a living, moving archive that, that constantly, you know, is a dynamic, collections of knowledge.There is a, there is a popular conception of use ??? and, you know, the stacks of this very kind of like, still space, like, you know, there's a real sort of sense of permanence and the idea that like everything is organised and that things move very slowly here. And so one of the things I think would be, like interesting to play with in terms of like how we use could use digital technologies to, play with the, the data set of, of a library of knowledge would be to try and think about ways of representing how those collections and archives have moved between areas of interest over time.And think about ways that we can create sculptures from that data that impress a sense of the dynamism that exists within, these, collections, but simply isn't observable on the timescale with which, like, we're used to observing things as individual humans.0:30:23 - 0:30:25
Paula Bray
Beautifully said.0:30:25 - 0:31:52
Paul Duldig
Beautifully said, but also makes it makes me think that one of the things that really weighs on our minds certainly weighs on my mind is this is a collection of active collecting over 170 years.But obviously the knowledge that's been collected goes back thousands of years, many thousands of years in some cases. And so we are custodians, for just such a short time of this incredible archive and it's very hard to, be able to take a third or fourth person view of that because you're in the middle of it. And, you know, you you think, well, in years to come, somebody will look back and say either ‘what on earth were they doing?’Or, ‘that was pretty good’. And so but but how do you how do you sort of, conceptualise that, in real time and, you know, there's, there's potentially ways of doing that through. Being more intentional about what your collecting practices are, what your biases are, how that's playing out, how that might map or not map against what people are actually looking for.Yeah. I mean, it's it's should be a tool of, you know, beyond an art piece to, to actually really understand what on earth we're doing and what they're actually doing a good job or not.0:31:52 - 0:33:33
Yoyo Munk
Yeah. I mean, I think that like, I mean the, the that last part about like what people are looking for, and I think is also a really interesting component of it because like, you know, libraries exist, you know, this is almost like a conversational exchange of knowledge, right?Like, I mean, it's, you know, rather than like, I mean, you know, if I go into, a natural history museum, you know, assuming that I'm going through the public exhibition, like, you know, this is a curated space where someone has gone through this painstaking process of putting artefacts out for, for exhibition and, but like, with a and but oftentimes when I go to a library, like, I have an idea of what I'm looking for.But I have this conversation with the librarian with the search system, like, you know, like there are these, like, this is an actually sociable process in terms of how I'm introduced to the information that I'm looking for. And like, one of the nice things about this is that, like, you know, oftentimes what I think I'm looking for is not actually what I want, right?Like, and but like that idea that, that the way in which we are collectively interfacing with libraries, the kinds of information that we are seeking from it, like this is the active process through which culture is expressing what it means to be important. And in any given point in time, like this is shaping like this is a reflection of the values that we hold collectively.And how those and, you given time, we'll see how those values are shaped over the course of history.0:33:33 - 0:33:35
Paul Duldig
Thank you so much. Yoyo.0:33:35 - 0:33:36
Yoyo Munk
My pleasure.0:33:36 - 0:33:37
Paul Duldig
Wonderful conversation.0:33:37 - 0:33:48
Paula Bray
Really great to have you join us in this conversation. I know that Paul and I have pretty much enjoyed everything that you said, and,yeah, thank you once again.0:33:48 - 0:33:49
Yoyo Munk
Oh this has been such a treat. Thank you very much.