ML-based Interactive Data Visualization System for Diversity and Fairness Issues

Abstract

As the recent developments of artificial intelligence, particularly machine-learning, impact every aspect of society, they are also increasingly influencing creative fields manifested as new artistic tools and inspirational sources. However, as more artists integrate the technology into their creative works, the issues of diversity and fairness are also emerging in the AI-based creative practice. The data dependency of machine-learning algorithms can amplify the social injustice existing in the real world. In this paper, we present an interactive visualization system for raising the awareness of the diversity and fairness issues. Rather than resorting to education, campaign, or laws on those issues, we have developed a web & ML-based interactive data visualization system. By providing the interactive visual experience on the issues in interesting ways as the form of web content which anyone can access from anywhere, we strive to raise the public awareness of the issues and alleviate the important ethical problems. In this paper, we present the process of developing the ML-based interactive visualization system and discuss the results of this project. The proposed approach can be applied to other areas requiring attention to the issues.

1. INTRODUCTION

Since AlexNet [1] won the ImageNet Large Scale Visual Recognition Challenge in 2012 by a large margin with an algorithm called deep convolutional neural networks, Machine Learning (ML), a subset of Artificial Intelligence (AI) technologies, has been rapidly applied in a variety of fields. From generating keywords of a given image or video [2] to making stories [3], ML technology has begun to influence our lives more than ever. Moreover, Google’s DeepDream [4] project in 2015 brought intriguing insights of how machines learn in the neural network learning model and inspired many artists to implement the latest ML technologies into their creative works. With this trend, many new communities such as Artists and Machine Intelligence, creativeAI, AI experiments, PAIR, and NeurIPS’s Machine Learning for creativity and design workshop have also been formed to support sharing new ideas in the new fields of AI + creativity.

However, as more artists use AI into their works, some issues of profound importance such as diversity and fairness are being revealed also in the creative field. Data-dependency is one prominent nature of the ML technologies differing from rule-based systems. The outputs of ML systems are totally dependent on the data used for training. This data-dependency could cause severe problems as demonstrated in the case of Microsoft’s chatbot Tay trained based on Twitter data as a racist [5]. Diversity issue may look less obvious than the above case, but lack of diversity in training data generates systems that amplify the social injustice existing in the real world. Diversity issues are also related to fairness issues. This particularly includes accessibilities to resources such as high- performance computing power, big data, and advanced technological knowledges as well as general issues such as gender, race, ethnicity, language barrier, etc. To address the issues, some artists or researchers have tried to make those advanced technologies become accessible to more people. For example, Gene Kogan has published his Machine learning for artists course materials online for free so that anyone can learn [6]. And Rebecca Fiebrink has developed and provided a ML software, Wekinator, for artists and musicians, for free [7].

In this paper, we present a new approach to mitigate the diversity and fairness issue in the emerging AI + creative field. Rather than resorting to education, campaign, or laws on those issues, we have developed a web & ML based interactive data visualization system on those issues of the new field. By providing the interactive visual experience on the issues in interesting ways as the form of web content which anyone can access from anywhere, we aim to raise the public awareness of the issues and alleviate the important ethical problems. In this paper, we present the process of developing ML-based interactive visualization system and discuss the results of the project.

What we present in this paper was actualized as an exhibition , which was main part of the 2018 Seoul Mediacity Biennale. In the project, we invited thirteen of the globally leading AI artists and exhibited not only their AI driven artworks but also their thoughts on ‘AI driven society’, ‘diversity and fairness’, and ‘augmented intelligence’ in the form of the immersive interactive visualization to investigate the diversity and fairness issues in the emerging creative field. As one of the first attempts for the problem, this research can be useful for future studies on finding creative solutions on the issues not only in the AI-based creative field but also in any area. The process that we propose in this research can be generalized to other areas where those issues exist.

 

2. RELATED WORKS

AI has been incorporated into the art practice with the development of computer technologies for a long time. AARON [8] is one of the first projects in the AI based art. It makes drawings autonomously without human intervention. Cohen began to develop the program in 1970’s and continued to develop it in 2000’s. More artists had begun to incorporate AI practices into cultural production in 2000s’. Mateas [9] proposed his hybrid art and science practice, Expressive AI, as a new inter-discipline of AI-based cultural production, combining art practice and AI-research practice. He described it by borrowing notions of interpretation and authorship from both art and AI research practice.

However, the recent rapid growth of AI-based arts is from the development of deep learning algorithms in mid 2010’s. Gatys, Ecker, and Bethge [10] introduced a deep neural network that can create artistic images of high perceptual quality. In their work, they created images that combine the content of a photograph with the style of well-known artworks using a convolutional deep neural network. The quality of the style transfer was photorealistic, which was not possible until then with other algorithms. Particularly, Generative Adversarial Nets (GAN) proposed by Goodfellow, et al. [11] sparked the AI-based arts. GAN allows the artists and musicians to synthesize photorealistic images and audios that sound similar to the ones composed by human composers. GAN was used in Pix2Pix algorithm (Isola et al. [12]). Pix2Pix was proposed as a general purpose solution to image-to-image translation problems. It allows artists to synthesize photos from label maps, reconstruct objects from edge maps, and colorize images from gray images. Another GAN based CycleGAN algorithm (Zhu et al. [13]) even allows artists to train the synthesis network without paired training data set. It makes it possible to synthesize an image of horses from an image of zebras, a photo of summer season from a photo taken in winter season, and a painting with well-known artistic style from a photo. All these new developments in AI have opened new possibilities in the art.

As AI technologies, especially machine learning algorithms that make models based on data, become pervasive including the art area, concerns have emerged that widespread bias in human society will enter the model. One main source of bias is lack of representation in data sets. Buolamwini & Gebru [14] reported that a face recognition system had much higher error rates for black women than white men. Another systematic bias can arise if training data contains human decisions that are biased. Angwin et al. [15] reported that the COMPAS system, which was developed to predict the possibility that a prisoner will commit a crime again if released, has a racial bias. It wrongly denied release to a black person compared to a white person with a higher probability. These reports show the importance of such issues in the new area based on AI.

 

3. INTERACTIVE DATA VISUALIZATION

Developing the new interactive data visualization system to increase the public awareness on the issues required careful design choices in many stages including whose data to collect, what questions to ask, how to collect the data, how to analyze the data, and how to visualize the data.

To select artists and researchers to participate in this research, we carefully reviewed most of the ML based works and the related articles presented at 2017 Ars Electronica festival [16], 2017 NeurIPS Creative ML workshop [17], and Google AI experiments [18]. The final participants list is as follows: Gene Kogan, Mike Tyka, Shinseungback Kimyonghun, Oscar Sharp & Ross Goodwin, David Ha, Mario Klingemann, Scott Kelly and Ben Polkinghorne, Lauren McCarthy, Lubba Elliot, Seung Joon Choi, Jihoon Jeong, Seungil Kim, and Sey Min. We selected them with two criteria: the impact of the works (how much the work is considered as important contribution to this area) and diversity (we tried to secure diversity of the group in nationality, race, and gender).

We asked the selected thirteen of the globally leading AI artists and researchers twenty-six questions on three subjects. We built a machine learning model to analyze the data by discovering the keywords and the relationships among the answers, developed a web-based interactive visualization system, and finally presented the results in the form of immersive interactive visualization to promote social discourses about the issues.

 

3.1 Data Collection

We asked the participants three sets of questions1. Each set of questions is related to each subject: ‘AI driven society’, ‘diversity and fairness’, and ‘augmented intelligence’. All the responses were collected via online before the Biennale was held.

The first question set on ‘AI driven society’ focuses on general understandings of the machine learning technologies in creative practices. The respondents shared what they have experienced while applying the advanced technologies into their creative works. One main question we asked under this subject is “Did you have any specific struggling point while working on AI driven projects?”.

The second question set on ‘diversity and fairness’ dives deeper into the problems that we are primarily interested in. The issue of diversity and fairness is related to all areas where AI is applied to. However, we focused on the issues only within the creative practice. Some of the questions we asked are “What are the ‘diversity’ and ‘fairness’ issues you have experienced in art and technology?”, “Have you taken any actions to reduce the imbalance or unfairness?”, and “What actions or resources do you think we need to solve the issues?”. The third question set asks the participants about ‘augmented intelligence’ [20]. This topic is related to the second topic ‘diversity and fairness’, but with more focus on intelligence. Some of the related questions are “Which group would get the benefits from augmented intelligence if not all?” and “How can we handle the unfairness situation?”.

We received total one hundred forty-one answers through Google forms via online survey. Some of their answers were in agreement, some were contradictory, and some answers were controversial.

On the questions of diversity and fairness issues, we could find a spectrum of interests and opinions. One said “Being a privileged white male, my problem is that I do not feel like I am the right person to discuss this. I try to increase diversity whenever there is an opportunity within my means, but I am not an activist or try to spearhead a movement.” while another participant answered “In both art and technology, people face bias and discrimination with respect to their varying features … These challenges are easier to face for people with comparatively more privileges like time, money, low predisposition to discrimination, and safety nets in case of adversity. People with a lack of privilege are more often forced out of the space, leading to homogenous culture and low diversity. For this reason, art and tech tends to be less diverse than other fields.”

One respondent pointed out that the idea of meritocracy is used as a justification for a lack of diversity in our societies while another respondent brought an interesting point in the imbalance saying, “I think education always exploits the unequal distribution of information and knowledge and passes over such imbalance to later generations.”. One AI curator said that sometimes the issues are created in the gallery by limited budgets.

We could also receive varied opinions on the issues of data dependency, fake and reality, bias in data sets, and inclusiveness. On data dependency, one pointed out that “Missing datasets mean that certain parts of the population will not be at all represented by a system.”, which is one important issue addressed very frequently in AI. On fake and reality problems, one respondent said “I was mostly still experimenting simply with the technology of creating machine that generates faces, but during that process I was thinking more and more what this sort of technology, accessible to anyone, will do to our world. The ability to fake identities, fake opinions, etc. … we're heading towards a time where it's going to get harder to know what's true anymore”. On the inclusiveness issue, one artist and educator answered “We are lowering floors by designing easier tools to work with AI. This is why more people work with AI, are interested in it, and know things about it, compared to 30 years ago. But at the same time, the technology is becoming more complex. Both of these patterns will continue into the foreseeable future.” Another respondent also said, “AI can primarily only amplify the intelligence of those who are technically skilled and/or have financial resources.” suggesting only certain people would get the benefits2.

 

3.2 Analysis

To find the patterns in the collected data, we used Word2vec [22] to reconstruct the linguistic contexts of the words in the answers. The corpus consisted of total 2951 words. Word2vec assigns each unique word a corresponding vector in a hyper-dimensional space such that words that share common contexts are positioned in close proximity to one another in the space. We used the Skip-Gram model to train the model. We also used tf-idf (term frequency-inverse document frequency) [23], a statistic that indicates how important a word is to a document in a collection of documents, in order to give weights to the words and produce better results.

To map the distribution of those words from the hyper- dimensional space onto two-dimensional space for visualization, we used t-SNE (t-distributed stochastic neighbor embedding) [24], which is a non-linear dimensionality reduction algorithm capable of capturing the complex relationship among features when mapped to a lower dimension better than a linear algorithm such as PCA (Principal Component Analysis).

 

Fig. 1. Part of visualization of the collected data after analyzing it with Word2vec and t-SNE. Words in similar contexts are positioned in close proximity to one another in the two- dimensional space.

 

3.3 Visual Interaction

Based on the analysis of the responses using word2vec and t-SNE, we implemented an interactive visualization system where users can explore the data in interesting ways. The purpose of this visualization system is to help the public understand the collected data in more intuitive ways and find answers for themselves to the questions “In the diversity and fairness issues in the emerging field of AI based creative practice, what is the current status and how can we improve the situation?”.

We implemented our visualization system using p5.js [25], a JavaScript library for web based interactive graphics, so that anyone can interactively access, explore, and reflect on the data on a web browser. People can interact with the visualization by a mouse. They can click on a word for investigation and scroll up/down to view overflowing content related to the selected word. When there is no user interaction, users view constantly changing sets of words positioned and moved on a screen in interesting ways.

When a user selects a keyword of his/her interest, the visualization system performs a series of actions. First, it rearranges other words around the selected one according to their proximity to the selected one based on the t-SNE output. Second, it shows the answer corresponding to the keyword with the respondent name and also top ten other related keywords.

Lastly, it displays the related question with more visual emphasis than others, which is designed to encourage users to ask themselves the same question while experiencing this system.

When there is no user interaction for a certain period of time, it goes back to the idle mode where users see all the available keywords placed and moved in a way that attracts users. Table 1. shows top five related words for some of the keywords that are of interest in this project3.

 

Table 1. Top five related keywords for some of the keywords of interests as analyzed on the collected data set.

 

Keywords The top five related keywords diverse less, fair, accustomed, starts, ideas fair desire, represent, starts, characteristics, mean fake identities, opinions, ability, already, analogy ai favors, artist, together, guiding, particularly data through, insights, training, distribution, outside The interactive web-based visualization is accessible at [26].

 

Fig. 2. The interactive visualization system. In idle mode, keywords continue to move in a way that attracts users.

 

Fig. 3. In the visualization, once a user selects a keyword (robots in the image), the related information (respondent name, other related keywords, and response) is visualized on the screen. The question (displayed left in the image) is displayed with more visual emphasis to encourage users to ask themselves the same question.

 

4. EXHIBITION

The proposed visualization system was exhibited as main part of the 2018 Seoul Mediacity Biennale. Along with the interactive visualization work described above, we have selected and exhibited the artworks of the participants that can be considered representative for this project.

 

Fig. 4. The 2018 Seoul Mediacity Biennale. (© Seoul Mediacity Biennale 2018, Photo: Cheol Ki Hong)

 

Mike Tyka exhibited his two project results on fake and reality. Tyka’s Portraits of imaginary People used a Generative Adversarial Network (GAN) to explore the latent space of human faces and generate the portraits of imaginary people, which raises the question of fakeness in the particular image domain. His other project, Us and Them, is an installation artwork also about the issue of fake and reality in the AI driven world. He used GANs to generate fake tweet streams. The machine learning model was trained on about 200,000 tweets from evicted twitter bots. The generated tweet streams trickled down from thermal printers at the ceiling, forming a central space. Over the course of the exhibition they slowly buried the two chairs at the center.

 

Fig. 5. Us and Them by Mike Tyka, installation, 2018. (© Seoul Mediacity Biennale 2018, Photo: Cheol Ki Hong)

 

Mario Klingemann created a new artwork, Uncanny Mirror, for this project. In the work, he also used a GAN to reconstruct a face from the model trained with the faces of its observers. Every day during the exhibition, the system recorded a set of facial features of the users who experienced the artwork during the daytime, and it updated the ML model with the newly recorded data at night. That made more exposed facial features become more strongly represented in the machine learning model during the exhibition. In other words, this artwork allowed us to more directly and visually observe the implication of data dependency of ML technologies.

 

Fig. 6. Uncanny Mirror by Mario Klingemann, interactive installation, 2018 (© Seoul Mediacity Biennale 2018, Photo: Cheol Ki Hong)

 

In addition, David Ha exhibited sketch-RNN, which is a project that enables anyone to finish his/her drawings with the help of ML. Gene Kogan presented Neural Synthesis, where he used the latent space inside Google’s Inceptionism network to synthesize a series of images. On the issue of fake and reality, Shinseungback Kimyonghun exhibited Nonfacial Portrait. In the work, they asked painters to draw a face that AI cannot recognize as a face. Oscar Sharp & Ross Goodwin presented Sunspring, which is an experimental science fiction short film based on the script entirely written by AI. And Scott Kelly & Ben Polkinghorne presented Signs of the Times, which shows how the Internet is affecting our lives by satirizing the biased recommendation system that we use every day.

 

Fig. 7. Signs of the Times by Scott Kelly & Ben Polkinghorne, billboards, 2018 (© Seoul Mediacity Biennale 2018, Photo: Cheol Ki Hong)

 

All the works from the artists and researchers who participated in this research were exhibited in order to introduce the new emerging field of AI-based creative works to the public and more importantly raise the diversity and fairness issue in the AI-based new area. Therefore, the exhibition culminated in our immersive interactive visualization system. The audience who experienced all the works from the participants were invited to experience the visualization system. They used a mouse on a small podium and explored the responses by interacting with the system. They could select keywords of their interests and find the related questions, responses, and other keywords in a visually attractive way. By doing that, they could not only find how the current practitioners are thinking on the issue, but also ask themselves the same questions.

Fig. 8. at the 2018 Seoul Mediacity Biennale. Viewers can interact with the visualization of the thoughts of the participants. (© Seoul Mediacity Biennale 2018, Photo: Cheol Ki Hong)

 

5. DISCUSSION

A total of about 66,000 visitors visited during the exhibition. During the exhibition, we held a public discussion, where the public and the artists participating in the project met, exchanged ideas, and discussed about the issues.

The issues of gender and racial diversity in particular have been discussed more. One audience pointed out that the fact that the AI home speaker is mostly set as a female character is solidifying the woman as a secretary role in the society.

Another audience also pointed out that the development of AI led by white men is further amplifying the white supremacy. To those comments, one panelist pointed out that there is always a tight tension between considering diversity and fairness and making the product sell, while another panelist pointed out that the gender and race issues are not from the flaws of AI technologies, but from the humans who train them, emphasizing the responsibilities of humans in solving the issues. The fact that most of the participants in this project were white men was also pointed out by one audience, and it ironically shows the current biased state of the technology driven ecosystem.

In addition, there has been a discussion about how to make this technology easier for more people to learn and apply to their own creative practices. As a ML artist and educator, a participant told that people in this field are constantly developing easier tools and learning materials so that more people can use it without a computer science background and that these efforts must continue. Another artist acknowledged that learning AI technologies, which are being developed very quickly, is a very tough process. He suggested that one should start by playing with the technologies, try various things, and ultimately has to find his/her own ways of learning.

 

Fig. 9. The panel discussion on diversity and fairness issues in the creative fields using AI. (© Seoul Mediacity Biennale 2018, Photo: Cheol Ki Hong)

 

In this project, instead of resorting to educational or campaign approach for raising the awareness of the ethical issues, we have developed an interactive visualization system and held an exhibition. Although the effectiveness of the proposed method is hard to quantify compared to traditional educational or campaign approach, the fact that more than 60,000 people visited the exhibition shows the impact that might have influenced the audience on the issues. Furthermore, this approach of collecting opinions from opinion leaders, analyzing based on machine learning algorithms, and presenting them in new and attractive ways including user interaction can be applied to other areas that require attention to a particular issue. However, the effectiveness of this approach will have to be further studied in more rigorous experimental set-up.

 

6. CONCLUSION

As AI is rapidly evolving and is expected to affect more and more of our society including arts, social problems such as diversity and fairness have become important problems that we have to solve. In this paper, we presented an interactive visualization system for raising the awareness of the diversity and fairness issue particularly in the emerging area of AI based creative practice.

In addition to developing the interactive data visualization system for the issue, this project has gathered ones who are currently leading the AI and creativity field and provided a forum for them to form a consensus on the seriousness of the issues and contemplate how to improve the situation along with the general public through an inquiry, data analysis, interactive visualization, exhibition, and public discussion.

To the best of the authors’ knowledge, this project is the first attempt to collect the thoughts of the leading people in this field and analyze and share them with the public on the internet through interactive visualization. In the future, more research is needed to find new interesting ways that can make people pay attention to the issues and take actions toward more inclusive cultures along with continuous education on the issue. As with all other technologies, how artificial intelligence technology will change our lives will depend entirely on us.