Artistic NLP Projects,

mostly using AI

An Introduction for Artists

from Google Arts & Culture Creative Lab

Lynn Cherny, Ph.D., ML Artist in Residence

Aug 2022

To view all slides, just press SPACE

my background bio

• Linguistics PhD from Stanford University
• Worked in user interface design (not visual, functionality!) and user research for 18 years at many well-known companies (including Adobe, Autodesk...)
• Occasional professor of data science (3 years at 2 universities)
• AI, data science, NLP consultant based in France
• newsletter on AI art tools/storytelling/NLP

Big Picture Topics

• Text Generation: Story vs. Poetry
  • Examples, story - AI and non-AI
  • Examples, poetry - AI and non-AI (data-driven)
• Multimodal - Images with Text
• Machine Translation
• Other NLP topics for art

Text Generation: Story and Poem

• Longer, complete sentences or dialogue
• Coherence over long sections
• Coreference - pronouns to names and entities
• Discourse structure: Arc of the story, rising action, etc.
• May mix dialogue and exposition
• Multi-characters/scene etc.

• Depending on type, may be "freer" or more restrictive than text
• Rhyme patterns are a constraint
• Metrical structure - iambic pentameter, haiku etc.
• Topical coherence may be less important
• Allow people to "read in" metaphor and synonymy etc.

Robin Sloan

Ken Liu

SF writer writes about using GPT-3 to write a story

How Novelists Use Generative Writing Tools: An Exploratory User Study

Writing with the machine  - first with RNN,  GPT-2

Some AI collaboration books exist... e.g.:

Pharmako-AI, 

By K Allado-McDowell (and GPT-3)

The Paranoid Transformer

by 

Aleksey @altsoph Tikhonov

You can try out text generation online:

AI Dungeon

NovelAI, SudoWrite

GooseAI Playground

GPT3 Playground/API - registration needed

Writing / using code in colab notebooks or github repos

Narrative Text:  Try It!

AI assisted writing startups, e.g SudoWrite

Our demo output btw :)

Story Telling Game Products: AI Dungeon, NovelAI 

NovelAI shown

Collaborative story generation with a model trained on genre fiction, or game play

Lots of Research on Story Generation!

See, e.g., challenges in Automatic Story Gen (article)

and my papers site on story (and other) generation research

Also lots of articles on text gen tools in general, e.g., A Robot Wrote This Entire Article. Are You Scared Yet, Human? (The Guardian)

And recent Verge piece on independent writers and SudoWrite / JasperAI

Tech Alert

GPT-3 / GPT-2 / Relatives (autoregressive language models)

 Text Generation via Procedural Methods

not AI models (but could be combined)!

"Slot" filling - "MAD libs"

Tracery, a tool by Kate Compton

TECH: Tracery.js

Dwarf Fortress Stories from Data Using Tracery

Interactive fiction game engines

Twine

Wayfarer

A Twine game

tools that help you write a branching game without coding

Another classic text "game"

Nested by Orteil, 2011 (thanks to Aaron A Reed for link)

NaNoGenMo

"National Novel Generation Month" (2021 link)

Write a novel using generative text, post the code and output. (Mix of AI and non-AI methods used.)

"Some Dim, Random Way"

Poetry Generation

First, AI and Machine Learning Poetry....

Many methods for poetry, too...

• Autoregressive Language Models like GPT2, GPT3
• Using Machine Learning or Embeddings as data input to poetry - providing words or themes
• Entirely procedural, using algorithms and choice lists

Language model approaches...

Google Experiments Link

Es Devlin at Serpentine Gallery exhibit, Simon Bird

TECH: Uses an LSTM model, 2016-17, trained on 19th c. poetry

Also in Lions in Trafalgar Square exhibit for London Design Festival 2018

Es Devlin's Installations of Generated Poetry

Es Devlin's UK Dubai 2020 (2021) Pavilion

TECH: Uses a GPT-2 model, much improved over LSTM, trained on 19th + modern poetry

Image/article source

GPT-2 Model training aside

We trained on poetry reformatted as:

<keyword>: 2 lines containing <keyword> somewhere

So that the model would take an input word from a person and generate a couplet containing it.

Ross Goodwin's captions

(using LSTM generation and image recognition)

Poetic "captions": generate poetry from images

code link & paper -

the code no longer runs

but it could be replicated with

current tools

Allison Parrish - other ML models

"Reconstructions" uses a VQAE model to generate "chiasmus"

Poety generated via image recognition - words provided for search / generation

TECH: Image recognition algorithms

word2vec / embeddings

 & other similarity methods

An embedding results in giving you words (or sentences, or images)  that are "similar" in some way, such as context of usage

Text

Demo of attributes

My link

Color represents a normalized distance between the original word (random "nouny"*-words) and its next closest relative in the embedding model.

Blue: Closer, Pink: Further Away. (So, options for "sound" are going to be more different.)

word2vec model:

pre-configured poms:

Click on a colored word like "sound"...  pick a replacement from it's nearest neighbors in the model....

Original haiku by Bashō:

tech: word2vec / embeddings

demo link

AI Poetry Research - More and More Languages and Forms

Chinese (Classical, Modern, Lyrics...), Urdu and Hindi, Finnish, Japanese, ....  (see my page here) 

Two specific links:

A good research article on training GPT-2 for poetry and limericks

Link to a good popular article overviewing attempts at generative poetry in research

"AfriKI: Machine-in-the-Loop Afrikaans Poetry Generation" - trained model on fiction by South African novelist Etienne van Heerden in Afrikaans to generate poetry (LSTM)

A person chooses and orders the lines generated, in a co-collaboration. Like the word2vec edits.

Sample online AI poetry generators

Boredhumans app (GPT-2)

Verse by Verse (Google)

RNN Markov generator in code

Other Digital Poetry Experiments

Based on data, interactions, etc.

Open Street Map Haiku (not AI) by Satellite Studio - based on data in the map location

Every Thing Every Time - Naho Matsudo (Manchester, Newcastle..)

article link

"Watch the live story of Newcastle unfold with an installation that turns city smart data into a constantly changing poem outside the Theatre Royal."

link

the web site is dead now, which makes me very sad.

JR Carpenter's "This is a Picture of the Wind"

Digitally remixed poetry, depends on the weather report (multi-award winning plus book)

Color Name Poems

by me

Ranjit Bhagnar's Pentametron, a twitter bot that got a lot of press in 2012+ for creating iambic pentameter poetry composed of found tweets.

E.g,

Gawker article

another article

Blackout Poetry

Lots of tools/libs online to make it, e.g.

Pre-AI Inspirations: Oulipo

One Hundred Thousand Billion Poems by Raymond Queneau

The book is a set of ten sonnets printed on card with each line on a separate strip. As all ten sonnets have not just the same rhyme scheme but the same rhyme sounds, any lines from a sonnet can be combined with any from the nine others, allowing for 1014 (= 100,000,000,000,000) different poems

source

(could be made with code easily)

Choose Your Own Adventure Poetry

Autocorrect poetry

Brian Bilston

Martha Silano

...

Google "search" poems

Brian Bilston

See https://www.reddit.com/r/googlepoems/

"Search" poems

Search for rhyming lines in a

corpus of poems / text

TECH: RiTa.js

"Search" by keyword & generation

input a word, get a couplet containing it, and build up a poem doing this.

Text

(my work)

TECH: ElasticSearch db

A Few Refs: Digital Poets

• Passage Tomb - Irish AI generated poetry and art
• Gwern's GPT poetry links
• Angel Carmona - Poemas V2
• Nick Montfort - curator/editor and experimental poet
• David Jhave Johnston - ReRites (collab with AI model)
• Sam Riviere and his press
• James Ryan's press Aleator - experimental works
• Sasha Stiles for the NFT Poetry movement and works
• Works in various digital experimental lit journals like Dead Alive and Taper
• For historical works, you can search the Electronic Literature Org's archives too

Multimodal AI - Images and Text Models

text2image and image2text, primarily

Text2Image Tools

Given a text "prompt", generate an image. You can now do this in Midjourney and at OpenAI's DALL-E for payment -- or use an online code notebook (Colab) that takes a long time to run.

OpenAI's

DALLE-2

Midjourney

Try yourself, in open beta now.

demo it.

Artists Are Using Text2Image Tools

• Comics / graphic novels
• Stories
• Animation
• Textures - Blender guy
• NFTs
• Character design / concept design for games, movies

Other Tools for Text2Image, Less Sophisticated

• Wombo.art
• Night Cafe
• Dalle-e Mini now at Craiyon (slowish)
• Stable Diffusion - just released code, in beta on Discord
• HuggingFace demos (free) but slow or limited
• In Google Colab notebooks - see my regular links in my newsletter on AI art

Multimodal artistic Search: Writing With Open Access - by Jono Brandel

Image2Text

• Captioning
• Question Answering
• Content / Poetry generation
• ....

A "novel" constructed of image caption/descriptions in a video

by trycatchfinally for Nanogenmo 2021

Story gen from Images

"MAGIC" Demo site

Telephone games

1. Starting prompt: "A lonely traveler walked by a lake"

2. Generate image using text with a code tool

From my Wordplay invited talk

How it starts

How it ends...

"Frankenstein's Telephone" by J. Rosenbaum

link, and see thread of recent attempts with non-binary persons imagined by Midjourney and Dalle-2... which follow:

The caption spawns an image, which is classified and divided into colored segments depending on what the neural network sees, another image is created using those abstract sections as a guide and a final caption is generated.

visualizing transgender and non-binary people

AttnGAN

2017

thread link

Joy Buolamwini: "AI, Ain't I A Woman?" (2018)

YouTube, text 

Machine translation

a very few examples, to stimulate some ideas?

"untranslatable"

Selections from IG poetry poster fossilisedflowers

Caroline Bergvall

Variation piece based on 47 translations into English of Dante’s opening lines of The Inferno.

Telephone Translation (mine)

listen: there is a good universe beside her; let's go
listen: there is a good universe nearby; go
listen: there's a damn good universe; gone
Listen: Next door is a very good corner. So let's go
Listen: The side door is a very good angle. Let's go
listen: there is a damn good universe; go
Please try listening. There is a very good universe
around. go
Please try to listen. There is a very nice corner on the
side. Okay, let's go
Listen: The side door is a very good corner. Let's go
Listen: there is a very good environment. Let's go
Listen: The neighbor has a very good angle. So let's go

English-Kurdish
English-Albanian
Kurdish-Finnish
Belarusian-German
Kurdish-Finnish
German-Albanian
Japanese-Albanian

Japanese-Kurdish

Somali-Kurdish
Haitian Cr.-Kurdish
German-Finnish

ee cummings   "listen: there's a hell of a good universe next door; let's go"

Mark Sample did with a book for Nanogenmo 2021

Tech: Translation APIs

Other NLP applications of possible interest

Dialogue Agents

"Sigana: Tales of Lawino"

Link

telling immersive VR folk stories with multimodal AI conversational agents

(work by Melisa Allela)

Multimodal VR Storytelling With Chatbots

Sentiment Analysis

"The winding geometry’s angle and its base ascent are respectively driven by sampling a sentence from the book and by performing a sentiment analysis on it."

"Stendhal", by Damien Seguin

Is it positive, negative, neutral; happy, sad, excited, etc....

Style Transfer of Text

See also, Exercises in Style, Raymond Queneau

Information Extraction

You might then use the facts in a new form, like a game!

Networks of relations in texts

Martin GrandJean

Character interactions in Game of Thrones

and Text Art

See my links on text vis & art

A Few Afro-centric NLP Links - See Jade's Talk!

Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Go (recent paper at ACL 2022)

Afrikaans poetry generation experiment

TextToSpeech dataset in sub-saharan african languages

An article on AI Art in Africa from AyaData by Luba Elliott

Sigana, Tales from Lawana by Melissa  Link

Thanks... and I'm here to help

If you have tech stack questions, I can point you to tools or methods!

Tech Notes on AI Text Gen Models

• "State of the Art" for long text generation are large models like GPT-3 (proprietary), GPT-J (open-source), NEOX...
  • In these models, you give a text prompt, and it "completes" it.
  • They are hard or expensive to "tune" on different texts; usually they are "prompt-engineered" with examples to steer them.
  • Awesome GPT-3 is a list of apps built on GPT-3
• Same class, but smaller, and more affordable: GPT-2. Can be fine-tuned and hosted for apps.  Not as high quality.
• If you just want single words, BERT/T5 etc. can be used. Those will give you missing words in a sentence.