Onirica () is an audiovisual work that explores the dimension of dreams, interpreting through synthetic languages the creative ability of the human mind during sleep. Through the use of algorithms capable of translating textual content into images, Onirica () brings tales of night visions back into the domain of the visible, proposing novel reflections on the relationship between human and machine, between tool and creator.

Dreams are experiences that have united and fascinated humanity since its origins. During sleep, our window to reality closes and gives way to a particular state of consciousness where thoughts and sometimes bizarre dream narratives follow one another, projected in our minds like cinematic sequences that are at times vivid and extremely defined. The stuff of dreams comes almost entirely from perceptions of the external world during wakefulness and exploits a reorganisation of memories that integrates experiences with fantasies, desires and more or less recurrent thoughts.

An increasingly common protagonist of scientific research, dreams are the subject of studies that have the goal to understand their features and characteristics. It is precisely thanks to the collaboration with two dream banks, the first from the University of Bologna and the second from the University of California Santa Cruz, that Onirica () came to life: through meetings with researchers, data were transformed into narrative elements, stories into visions, elaborating a project that would relate the scientific method to the fluidity and creative mutability of oneiric activity.

The work transforms into a collective experience the dreams of volunteers who participated in research sessions at the two universities. Selected from a base of 28,748 dreams, the plots flow one into the other as a series of short films, tracing the actual cadence of NREM and REM dreams present over the course of a night's sleep. The sequences are artificially generated by a machine learning system that translates the text of dreams into a series of subsequent hallucinations that bring to life the characters, objects and landscapes described.

This continuous synthetic stream of consciousness finds its final aesthetics through the close collaboration between human being and artificial intelligence: while the machine proposes endless possible translations of the stories into images and voices, it does not possess any kind of decision-making ability concerning aesthetic and conceptual choices. The technology thus assumes the role of a creative assistant that interprets directorial directions by proposing possible ideas and solutions, in a relationship comparable to that which develops within a film crew composed, in this case, of humans and intelligent machines.

Onirica () accentuates the tension created by the interpretation and translation of a purely human experience, the dream, through the eyes of new technologies. Inserting itself within an increasingly relevant ethical debate, the work aims to address from an unprecedented and exploratory point of view the relationship between a purely human sensibility and the creative capacity of artificial intelligence systems: to discover their potentialities and limitations, to stimulate in the viewer a critical and conscious thought about the possible impact of these technologies on society and on the perception of ourselves.

DREAM BANKS

Onirica () developed from a database of reports of 28,748 dreams: of these, 807 were collected and transcribed anonymously in the Laboratory of Psychophysiology of Dream and Sleep "M. Bosinelli" of the Department of Psychology "R. Canestrari" of the University of Bologna (Dream Data Bank, DDB), and 27,941 collected in the Dream Bank, a collection conceived, designed and built by Professors G. W. Domhoff and A. Schneider of the University of California Santa Cruz.

In the first case, dreams were collected between 1970 and 2005 during specific nocturnal research sessions. These took place in specialised psychophysiology laboratories, consisting of soundproofed rooms for sleeping subjects and a room dedicated to the overnight work of researchers. During the nights, participants (mostly volunteer students in the case of Bologna) slept in the rooms after electrodes were applied to their bodies to record polysomnography (PSG) data, including an electroencephalogram (EEG), an electro-oculogram (EOM), and an electromyogram (EMG). These parameters serve to highlight the sleep stage the subject is in at a given moment: sleep is divided into NREM (non-REM) stages, which in turn consist of stages 1, 2, 3, and 4, and REM (Rapid Eye Movement) stage, characterised by rapid eye movements and muscle atonia.

Throughout the night, the volunteer would be awakened one or more times by an acoustic signal and asked to recount in as much detail as possible what they were dreaming at the time of awakening. The account would be transcribed verbatim, including any uncertainties, interruptions or repetitions. Only after the completion of the free narrative would any specific questions be asked in order to categorise the dream within precise parameters, such as first or third-person perspective, well-defined setting, number of individuals, presence of oddities, presence of dimensional or space-time distortions. These characteristics are also used later for dream categorization.

The focal point of the research nights at the University of Bologna was the study of dreams as a cognitive process that occurs during sleep. For this reason, dreams were catalogued with the aim of understanding the sources of memory from which the mind draws for their generation. Following the mentioned research, it was demonstrated that dream activity is present during all stages of sleep. The main differences observed between phases are the higher occurrence of dreams in REM stage (90% of awakenings in this condition) compared to NREM conditions (50-70% of awakenings). Furthermore, dreams in REM stages are longer and somewhat more intricate than those collected in NREM stages. 

The second Dream Bank was created by psychologists G. W. Domhoff and A. Schneider from the Department of Psychology at the University of California Santa Cruz and represents one of the most important scientific contributions on the topic. This database currently contains a collection of 27,941 accounts from individuals aged 7 to 74 years, sourced from a variety of different contexts and sources—making it distinct from a specific research or analysis. The database is accessible online, making it easy to consult for both scientific and informative purposes. The website allows for simple keyword searches or more complex searches, such as cross-referencing various data to discover the likelihood of the presence of two specific elements in a set of dreams.

Scientific research in the field of dream material remains complex for several reasons, primarily due to the impossibility of verbal reports on the dream process as it occurs and the consequent need to base studies on written testimonies subsequent to the dream experience itself. Additionally, navigating dream memories and relating them to each other is challenging. This inherent complexity necessitates the creation of easily accessible and usable systems that allow for the exploration of differences and similarities in dream content on cross-cultural, gender, and individual levels. The work of Domhoff and Schneider differs significantly from that of the University of Bologna but is valuable for providing a second perspective on the relationship between dream experiences and wakeful states. In the article "Studying dream content using the archive and search engine on DreamBank.net," published in 2008 in the journal Consciousness and Cognition, the authors discuss the results of scientific research based on dreams from the Dream Bank. An interesting aspect that emerges from some studies is that when analysing dreams, individual differences appear to be less relevant than commonly believed.

USE OF DATASET & DREAM SELECTION 

Onirica () was born from the narratives collected from a total of 28,748 dreams described upon waking by dreamers. The scale of this vast collection of data makes it opaque, difficult to explore by manual search. To recognize patterns, recurring themes, conceptual or semantic connections that link dreams even when distant in time and space, it was decided to navigate this space through an automated text analysis system. The text of each dream is analyzed by a Machine Learning (ML) model, which associates each of these with a point in a 384-dimensional space: the greater the affinity between the thematic or semantic content of two dreams, the greater their closeness. Through a dimensionality reduction method, the dream space was transposed into three and two dimensions, generating a dream map that can be read and interpreted by the human user.

The narrative of Onirica () was constructed from the selection of a defined number of dream memories within the dataset. Thirty dreams, peculiar in terms of richness of visual elements, bizarreness, level of involvement in the narrative, presence of engaging and imaginative themes, unrealistic visions, and subversion of physical laws, were chosen from the 807 accounts of the Laboratory of Sleep Psychophysiology at the University of Bologna.

In addition to the selection of narrative material, the second element that guided the construction of the work was the analysis of the architecture of dream activity. Sleep, like most human activities, is cadenced by a rhythm influenced by complex biological processes. Within a night there are cycles lasting about 90 minutes, alternating between non-REM and REM stages. In the first half of the night, most of each cycle is characterized by deep NREM sleep, and less REM sleep; in the second half of the night, however, this balance reverses, and most of the time is dominated by REM sleep.

The compositional structure of Onirica () thus echoes the actual alternation of NREM and REM dreams present in the course of a night's sleep, distributing the thirty dreams cataloged by the Laboratory of the University of Bologna into five cycles of six dreams each, forming the narrative basis of the entire installation.

The transition between contiguous cycles is represented by what we call bridges, which introduce a semantic journey that cross the database through similar dreams, leading to visualising - through a chain of adjacent words in space - the path between the last word of the sixth dream of one cycle and the first word of the first dream of the next cycle. Material from the DreamBank at the University of California Santa Cruz was used to create the bridges due to its composite and heterogeneous nature, which is essential to enhance the variety and vastness of the data complex.

In parallel with the visual narrative of dreams shown on the central canvas, additional textual content is projected on side supports in order to explore and expand the perception of the entire archive of collected dreams. Throughout the experience, simultaneously with the utterance of specific keywords, texts of other dreams in the archive, syntactically connected to these same words, are displayed on the side walls.
The same correlation between keywords and lateral projections is further explored during the bridge phase - which serves as an interlude between the 30 dreams - where the fast-paced rhythm of the words displayed on the central dream visualisation causes the texts on the sides to alternate very quickly, giving the viewer a sense of the thematic recurrence, complexity and size of the archive at hand. 

THE VISUAL GENERATION 

Underlying the visual development of Onirica () is a text-to-image diffusion model: a generative model that exploits neural networks to learn how to synthesise images from textual descriptions. The first step toward generation was textual analysis of the thirty selected dreams and their semantic structure. Each account was deconstructed into paragraphs, sentences and visual images and then processed through a Large Language Model (LLM), a Machine Learning model capable of working with natural language, used to extract the structure and visual information contained in the individual dream accounts. 

We then proceeded to write short texts (prompts) containing precise compositional and subject descriptions capable of directing the text-to-image model toward a visual translation of the textual content of the dream. The syntax used in the prompts is aimed at placing greater emphasis on the emergence of the main concepts and subjects in the generated image and to overcome the interpretive biases inherent in the algorithm used. Parallel to the individual compositional fragments, the experience in its totality is described and addressed by overall prompts, which define its aesthetic and sensory atmospheres that contribute to unifying the entire dream stream.

Just as dreams are characterised by more or less coherent streams of images, Onirica () is developed through a dynamic succession of visualisations very similar to that of our mind, referring back to the stream of consciousness that is generated simultaneously with its narrative. The visual flow that characterises the work was realised thanks to a custom pipeline that weaves together three main levels: a generative system based on particle systems, the text-to-image diffusion model, and its subsequent visual reworking. All synthesised into a single feedback process that advances over time and is influenced by the sound component, dream texts and associated parameters.

The particle system is responsible for generating the initial image; this is subsequently reinterpreted by the Diffusion Model on the basis of the dream text and through the management of some characteristic parameters (e.g. prompt embeddings, guidance scale, strength, etc.). The versatility of the Diffusion Models allows for a dynamic departure from the source image by giving greater or lesser relevance to the interpretation of the dream text: precisely the management of this balance turned out to be one of the most interesting expressive tools on which the entire visual development of the project is based. Finally, the image returned by the diffusion is modified by the third layer that deals with the introduction of additional compositional elements into the image. This last layer makes it possible to increase the overall compositional complexity and, through the feedback process, generate the basis for the next frame.

THE SOUND AND VOCAL COMPOSITION

The sound atmospheres of Onirica () trace the ideal pattern of sleep during the night through three distinct musical moments: deep sleep (NREM), REM sleep and the moment of transition between different cycles. Alternating between crescendo and decrescendo, the opera's soundtrack creates a transition between regular and irregular elements, elements of tension and relaxation, which follow one another tracing the rhythm of the narrative.

A drone, a technique whereby a note or group of notes are played continuously, has been used to represent sleep. A common use of this technique is in traditional Indian music, where it is achieved through instruments such as the tanpura. Deep sleep is also often referred to as "slow wave" sleep because neuronal activity slows down and tends to synchronise. Similarly during installation in NREM phases, the drone dries out by thinning out to a few harmonies and working at lower and lower frequencies.

Although it may be thought that during sleep brain activity is minimal, it is scientifically proven that during the REM phase the brain conducts neuronal activity comparable to or, in some specific areas, even greater than during wakefulness. In the sound composition of Onirica (), this electric "buzz" is achieved by using three instruments (electric bass, electric guitar and acoustic guitar) tuned to open tunings and played through a bow. The reiteration of this gesture contributes to the creation of a rich mantle of sound made up of variations and micro activity. In the bridging phase between one cycle and the next, which represents a true crossing of the dream space, the same timbres are explored through some extended techniques-unusual, nontraditional approaches-that expand the expressive technical possibilities of percussion instruments and strings, played along unconventional angles, particularly on headstock and bridge.

Another element of paramount importance in the experience are the voices, which accompany the visitor on the journey through the dreams: if in each bridge the voices mingle in a dreamy chatter, in each dream short film it is a single voice that is in charge of the main narrative. The thirty dream texts selected from the Bologna dataset by their nature (due to the fact that they are a verbatim description in the laboratory, collected at the end of the dream), also include special elements such as hesitations, repetitions, disconnected phrases, or interjections, which contribute to making the dataset even more real and human.

In order for this level of detail to be fully exploited, an artificial voice generation was opted for, implemented through an open source artificial intelligence model named Bark. Although the model occasionally succeeds in correctly interpreting texts through emphasised words, stumbles or sighs, it is equally interesting to be aware of its difficulty in exposing texts consistently and with the right intonation. For this reason it was necessary to do a particularly thorough and detailed job of generating multiple vocal traces, which have then been listened to and carefully selected. At times, it also required some manual cut and sew in order to merge different generations of the same dream.

CREDITS

Onirica () is a site-specific audiovisual installation produced by fuse* with the support of INOTA Festival and Fondazione Alberto Peruzzo.

The dream stories used in the artwork come from the following datasets:

• DreamDataBank (DDB) Laboratory of Psychophysiology of Dream and Sleep "M. Bosinelli," Department of Psychology "Renzo Canestrari," Alma Mater Studiorum University of Bologna.  With special thanks to lab leader Miranda Occhionero.
• DreamBank University of California Santa Cruz, created by G. William Domhoff and Adam Schneider
  https://www.dreambank.net/

The visual component is based on a pipeline integrating the Diffusers: state-of-the-art diffusion model library developed by Huggingface and OpenGL Shading Language (GLSL).
Connections between dreams were obtained through text analysis with the Sentence Transformer framework, first introduced in the article "Sentence-BERT: Sentence Embeddings using Siamese BERT- Networks," by authors N. Reimers and I. Gurevych.
The speech synthesis was realized thanks to the Bark model developed by Suno AI.

Photo report: Ugo Carmeni, 2023

Current

9 February - 28 July 2024 - "Ventanas al futuro" c/o Espacio Fundación Telefónica / Madrid, ES
27 January - 12 May 2024 - "Hello, Human!" c/o MoCA Taipei / Taipei, TW

Past Exhibitions

15 - 20 April - Pasqua Winery / Verona, IT
07 - 12 November 2023 - Taiwan Creative Content Fest / Taiwan, TW
16 September - 15 October 2023 - Fondazione Alberto Peruzzo / Padua, IT - solo exhibit
31 August - 03 September 2023 - INOTA Festival / Várpalota, HU