I recently started to use an emoji of a book in the captions of my paintings for my art project at the Centre for Renaissance and Reformation Studies at the University of Toronto, titled “Word and Image Relationships at the CRRS Library.”
Traditionally, we see images and captions as counterparts, where the image provides a certain view, and the caption provides information about this view or the artwork itself. If you look above, you see that the first line of the caption indicates the artist, the name of the artwork, its year of production, artistic technique, and current location. This is a conventional form of a caption. In principle, the role of the caption is to contextualize the image through minimal information; however, it can also be a space for creativity and play. Let’s answer two questions to better understand this possibility.
What is the role of an emoji of a book in the caption above? First, as you can see, the emoji is of an open book; neither of a close one, nor of a pile of books. It is important because the painting is made on a printed scan of a double spread, on a “recorded glimpse” of an open book. Thus, the emoji shows the “canvas” of the painting, although in this project, I paint on double spreads that include at least one image, and in this emoji, there is only text.
Second, the emoji starts a new line in the caption. On its right, there is an entry of a bibliographical list, which we usually find at the end of a scholarly work. Here, we can interpret the emoji of the book as a visual affirmation that the bibliographical entry is indeed about a book, even if it is written in a caption of an image. By denoting both the physical base of the painting (a book), and the use of a specific book, the emoji raises the awareness of the choice to paint on books, on the one hand, and to paint on a particular book, on the other. Introspectively, it also points out the use of an emoji to do so.
What is the effect of using an emoji in a caption of an image? In addition to the meanings the emoji may suggest, if it is colourful, for example, its presence seems to add “life” to the caption, just as colourful illustrations do in textbooks. This, in turn, increases the ability of the caption to pull the viewers’ attention toward itself. Does this affect the attraction of the image? Among other factors, it depends on the kind of emoji we use (book, heart, moon), and where we locate it in the caption (beginning, middle, end). To ensure that the emoji adds value to our work, we need to approach the latter holistically; to recognize that, beyond any categories, each of its elements is the preparation and the result of the other at the same time.
Apparently, today is Google’s 23rd birthday. There are so many reasons to wish Google a “Happy Birthday!” as it adds tremendous value to our culture and society. At the same time, I ponder where this culture of collecting and selling data goes. I wish I could get from Google and other companies the great products and services they develop without being constantly registered and targeted. Imagine the internet as a flea market.
At any rate, I want to write about the image that Google posted today on the Search homepage, showing an animated two-tier birthday cake, with chocolate icing, colourful candies, and a candle (see above). On the second tier of the cake, they wrote “23” to indicate the number of years the company has been around. The letters in the inscription “Google” above the cake look like cookies or doughnuts, and they are covered by chocolate icing and candies, just like the cake. Sweet.
What caught my eye this morning was the cake’s freckles that are spread on its face on the first tier. Since the chocolate icing above the face looks like the cake’s hair, we cannot identify the brown dots on the second tier as freckles as well. These and those on the “Google” inscription are probably crumbles. I find the use of the brown dots as both freckles and crumbles charming, and would like to take the challenge of explaining how this charm is constructed.
First, note how the pattern in which the brown dots are spread becomes less ordered as we go up the cake. The freckles are divided into two groups of five on each side of the face, while the crumbles on the second tier cover it somewhat equally, and the crumbles on the inscription are independent of any grid. Second, this gradual three-fold change in the pattern of the dots takes place in a small space. It means that the illustration requires us to adapt our interpretation of the dots quickly according to their pattern. What we think we see promptly becomes something else. This is how the wonder is constructed. The charm is introduced with the cake’s smile that is pleased with its wonder.
The following isn’t an answer for the question positioned above, but rather several thoughts about the use of both words and images in a document. I’ll analyze an invitation that I recently prepared to my upcoming hands-on workshop at the Koschitzky Centre for Jewish Studies at York University. The team at the Centre offered improvements to what I initially sent them, and we sent it today over relevant list-serves. These are the image and the text:
“Artist in Residence” is an art project Dr. Noa Yaari created at home for the Koschitzky Centre for Jewish Studies. Dr. Yaari likes to say that the project is about how she has experienced the current pandemic, but it is much more than that. In this hands-on workshop, she invites us to dive into its details and the deliberations that determined its look. For instance, we will focus on the elements that convey discontinuity and insecurity, on the one hand, and those that express optimism, and self assurance, on the other. This would be an opportunity for us to examine how works of art open spaces for personal interpretations but also utilize perceptual and cultural codes that we share. We will also participate in a 10-minute drawing exercise to unleash our creativity in a supportive environment.
First, I like to direct your attention to the terms “residence” and “home” and the image of the upside down house. The title of the art project, “Artist in Residence,” suggest that its theme is an artist being in residence, which is confirmed by the indication that the artist created the project “at home.” This, in turn, brings the meaning of “residence” and “home” closer, blurring the distinction between the professionalism of “residence” and the intimacy of “home.” The text argues that I like to say that its about my “experience of the current pandemic.” Indeed, one of the main characteristics of it has been the integration of work and home through the “work at home” model. Considering this, what does the image of the house contribute to the invitation?
If the artist created the project at home, and the image of the house is of the artist’s home (we know it was painted there), then the artist experiences a downfall. The text points out that I invite the audience to dive into the project’s details; their planning and the contradicting emotions they evoke. The use of the term “dive” emphasizes the movement of the house (or artist) downward, and the term “invites” legitimates any interpretations that the viewer may realize in their mind while reading the invitation.
Second, the text indicates that we will examine how artworks function on the fine line between personal interpretations and shared perceptual and cultural codes. This reminds the reader that their interpretation – perhaps association between – the terms “residence” and “home” and the image of the house may be idiosyncratic, and equally so, a collective reaction.
This verbal-visual composition intentionally creates a mental space that expands in various directions. It positions words and images in vicinity to one another to raise different meanings that make sense to the same degree, as paradoxes do. The hearts, roof, and snow in the painting enable the colour red to do so as well.
An articleon the RAND website explores the use and effectiveness of art-based approaches to public engagement with research. It argues that:
“When designing arts-based engagement approaches, it is important to consider that they can bring out experiences in stakeholders that may be challenging to process at an emotional level. Mitigations and management mechanisms to deal with potential unintended consequences should be included in the approach.”
The brief article from which I quoted this conclusion, doesn’t give examples. Thus, I can only guess what the author means by “unintended consequences.” What could they be? I recall myself being engaged with artworks of various kinds and reacting to them emotionally. I don’t remember myself experiencing “intended consequences;” they all seem independent of my wish whether to experience them or not and, in my memory, I’m fine with that.
The assumption that “unintended consequences” should be “managed” suggests that they entail a negative feeling, or perhaps a positive one, which may be inappropriate. Am I right? Otherwise, what could be the reason to manage them? Perhaps these consequences are sadness, anger, fear, disappointment, agitation, and other kinds of pain. If the results of research are painful, and the art-based approach delivers them in a creative and engaging way, what is the problem with experiencing that pain? Isn’t that unpleasant experience a condition for further ideas, conversations, activism, and change?
Lastly, the recommendation to include “management mechanisms” in the art-based approach raises the questions: What are these mechanisms? Who should design them; the researcher, artist, or other kind of expert? And would it be risky to trust the audience that it can deal with its emotions as it encounters reality, research, and art?
I’m currently participating in a course in Knowledge Mobilization (KMb) at York University, which is run in collaboration with the University of Winnipeg and Memorial University. Titled MobilizeU, it’s excellent. One of the things I appreciate most in it is the opportunity to delve into “Art-based knowledge mobilization” (ABKM) and its potential to engage diverse audiences with academic research.
KMb is a field that examines and develops methods of sharing academic research with non-experts. These can be academicians from other fields, organizations, communities, and the general public. KMb uses communication methods such as describing research in clear and engaging language, visualizing information, and disseminating research on social media. It aims to enlarge the audience of the research as well as to enable stakeholders to become co-producers of its impact. Hence, collaborating with community members, industries, and policymakers is essential to KMb.
For researchers, moving from a professional discourse to engaging non-experts with their research may be challenging. It requires the development of new habits, which entails learning and creativity. The expectation of research funders to see researchers involved in KMb activities adds pressure to that challenge.
AKMB, which is also called art-based knowledge translation (ABKT), offers solutions to making research accessible to broad audiences, and it’s also fun. For example, as a Fellow at the Centre for Renaissance and Reformation Studies (CRRS) at the University of Toronto, I create an art installation in the hallway that leads to its library. For the installation, I paint on 30 double-spreads that I scanned, which include both words and images and which I found inside the books of the CRRS library. While this project continues my academic research on the creation and communications of knowledge through combinations of words and images, it provides the viewers a sensorial experience and several “visual jokes.”
The freedom that art instills in research through KMb activities may effectively connect diverse audiences with the content of the research. One thing to consider is the format; it can be any kind of architecture, rather than a book or journal article. The physicality of the space and the artworks, on the one hand, and that of the viewers, on the other, can turn the knowledge into sensorial vibes.
Combinations of words and images have a ripple effect that includes attracting attention to themselves, retaining that attention, penetrating to and residing in people’s long-term memory, and stimulating the echoing of the stored information. Combinations of words and images are, therefore, seeds that often grow into a culture.
This ripple effect is a result of the forces that we generate when we position words and images one next to the other. Every time we bring the two kinds of signs together, we create a “magnet” between them, that not only establishes their mutual attraction but also attracts the viewers’ attention to the new hybrid composition. Furthermore, there is a correlation between the relationship between the words and the images within the composition, on the one hand, and the strength of the latter to attract and retain attention, on the other.
For example, if the words signify the past and the images allude to the future, the whole combination opens space for the audience to immerse in a creative interpretation. As if the audience’s role is to fill in the gap between the words and the images and, concurrently, between the past and the future. This process gives space and time for individuals to become part of the creation of the content. As their engagement lingers, this process also amplifies their self-awareness.
Diving into the interpretation of any content increases the chances of creating a representation of it in the long-term memory. However, there is a difference between trying to understand what something means and what someone says. The first may introduce an abundance of equally reasonable possibilities, whereas the latter may reflect a lack of clarity or unintentional vagueness.
Exploring and sorting possibilities while looking at visually appealing content is a pleasure, which in turn increases the will to continue consuming the content. Another form of consumption is to share the memory of the content and the pleasure associated with it with others. This level of the ripple effect gives even more space for the audience to re-create the information. At this point, the original combination of words and images is only a seed, as the rest is already a culture.
Every combination of words and images includes both repetition and variation. For example, when we use a letter, symbol, word, or phrase more than once, or when shapes and colours reappear. Some of the repetitions result from an echo between the words and the images, which may express different kinds – as well as degrees – of similarity.
In the image above, the letters C (c) and O (o) in “Commander,” and the circular shapes in the image of the ship, maintain similarity. They echo one another as if they have shared ancestors, culture, history, something beyond the division of “verbal” and “visual.”
What does this affinity between words and images or their elements do to our communication? What does it do to us? One possibility is to ask whether the letter O (o) could have more in common with a window of a ship than with the letter E (e), which may share the same word with O (o). Another possibility is to harness letters such as O (o) and windows of ships to increase cohesiveness in a hybrid environment such as verbal-visual communication. This approach to repetition would surely see the value in the variation.
Noa Yaari, CRRS 7/30 (detail), 2019. Mixed media. 21 x 28 cm. Toronto
I recently wrote an email to a friend of mine in which I used : ) Since I have Grammarly as a plug-in on my Chrome, I’m able not only to see the level of my grammatical performance but also the tone of my communication. Grammarly’s emoji reacted seriously to that email; their eyes were open, and their mouth was flat. Those of you who know Grammarly might have already guessed: it was the “neutral” one.
But, I thought, I used : ) in my email. How can Grammarly look so neutral? Can’t they see that I’m quite happy, and that I’m ensuring that my reader receives the right tone of my massage? While asking these questions, my first step was to sound more cheerful. I changed my wording until Grammarly smiled along with my smiley. And my second step was to look for information about what determines Grammarly’s moods or reactions.
The Grammarly website says: “Grammarly’s tone detector relies on a combination of rules and machine learning to identify signals in a piece of writing that contribute to its overall tone. By analyzing your word choice, phrasing, punctuation, and even capitalization, Grammarly can tell you how your message is likely to sound to someone reading it.”
Apparently, Grammarly’s analysis doesn’t recognize : ) as it computes features such as wording, phrasing, and punctuation. Let’s try to make sense of this technological, and ultimately cultural, phenomenon.
Grammarly uses an emoji to signal to the users of the application how they might sound to their readers. It means that while the application’s input is verbal, its output is visual. In the spacetime between the input and the output, the verbal information changes into a visual one. In principle, if the words sound serious, Grammarly looks serious. And if the words sound exciting, Grammarly looks excited.
Grammarly’s output in the form of a neutral emoji in my email tells me that the application is “blind” to the icon : ) Moreover, further examination has shown that it doesn’t recognize emojis that Gmail offers as well. Lastly, the company’s website doesn’t mention emojis as a variable that the tone detector processes; this confirms my observations.
How is it possible that Grammarly uses emojis to denote tones while it ignores them at the same time? It makes sense to me that if Grammarly “believes” in emojis, it has to “respect” them equally to words. Until then, I’m committed to making Grammarly’s emoji happy. In my emails, my friend, no one leaves you neutral : )