Artistic NLP Projects,

mostly using AI

An Introduction for Artists

from Google Arts & Culture Creative Lab

 

Lynn Cherny, Ph.D., ML Artist in Residence

Aug 2022

To view all slides, just press SPACE

my background bio

  • Linguistics PhD from Stanford University
  • Worked in user interface design (not visual, functionality!) and user research for 18 years at many well-known companies (including Adobe, Autodesk...)
  • Occasional professor of data science (3 years at 2 universities)
  • AI, data science, NLP consultant based in France
  • newsletter on AI art tools/storytelling/NLP

Big Picture Topics

  • Text Generation: Story vs. Poetry
    • Examples, story - AI and non-AI
    • Examples, poetry - AI and non-AI (data-driven)
  • Multimodal - Images with Text
  • Machine Translation
  • Other NLP topics for art

TECH ALERTS - If you want to know more later...

"Tech Topic"

Text Generation: Story and Poem

 

Story vs. Poem, first...

Narrative    vs.     Poetry

  • Longer, complete sentences or dialogue
  • Coherence over long sections
  • Coreference - pronouns to names and entities
  • Discourse structure: Arc of the story, rising action, etc.
  • May mix dialogue and exposition
  • Multi-characters/scene etc.
  • Depending on type, may be "freer" or more restrictive than text
  • Rhyme patterns are a constraint
  • Metrical structure - iambic pentameter, haiku etc.
  • Topical coherence may be less important
  • Allow people to "read in" metaphor and synonymy etc.

 

Robin Sloan

Ken Liu

Some AI collaboration books exist... e.g.:

Pharmako-AI

By K Allado-McDowell (and GPT-3)

The Paranoid Transformer

by 

Aleksey @altsoph Tikhonov

 

You can try out text generation online:

 

AI Dungeon

NovelAI, SudoWrite

GooseAI Playground

GPT3 Playground/API - registration needed

Writing / using code in colab notebooks or github repos

 

Narrative Text:  Try It!

AI assisted writing startups, e.g SudoWrite

Our demo output btw :)

Story Telling Game Products: AI Dungeon, NovelAI 

NovelAI shown

Collaborative story generation with a model trained on genre fiction, or game play

Lots of Research on Story Generation!

See, e.g., challenges in Automatic Story Gen (article)

and my papers site on story (and other) generation research

Also lots of articles on text gen tools in general, e.g., A Robot Wrote This Entire Article. Are You Scared Yet, Human? (The Guardian)

And recent Verge piece on independent writers and SudoWrite / JasperAI

Tech Alert

GPT-3 / GPT-2 / Relatives (autoregressive language models)

 Text Generation via Procedural Methods

not AI models (but could be combined)!

"Slot" filling - "MAD libs"

Tracery, a tool by Kate Compton

Rules define combinations (see "sentence" above) and the output can be many variations of these words.

TECH: Tracery.js

Dwarf Fortress Stories from Data Using Tracery

a project of mine... written up.

Interactive fiction game engines

A Twine game

tools that help you write a branching game without coding

Another classic text "game"

Nested by Orteil, 2011 (thanks to Aaron A Reed for link)

NaNoGenMo

"National Novel Generation Month" (2021 link)

 

Write a novel using generative text, post the code and output. (Mix of AI and non-AI methods used.)

 

Related but professional: A procedurally generated book in which no two copies are the same: Subcutanean by Aaron A Reed

"Some Dim, Random Way"

Kevan Davis's entry of Moby Dick as a CYOA game (link)

Poetry Generation

 

First, AI and Machine Learning Poetry....

 

Many methods for poetry, too...

  • Autoregressive Language Models like GPT2, GPT3
  • Using Machine Learning or Embeddings as data input to poetry - providing words or themes
  • Entirely procedural, using algorithms and choice lists

Language model approaches...

 

TECH: Uses an LSTM model, 2016-17, trained on 19th c. poetry

Also in Lions in Trafalgar Square exhibit for London Design Festival 2018

Es Devlin's Installations of Generated Poetry

Es Devlin's UK Dubai 2020 (2021) Pavilion

TECH: Uses a GPT-2 model, much improved over LSTM, trained on 19th + modern poetry

GPT-2 Model training aside

We trained on poetry reformatted as:

 

<keyword>: 2 lines containing <keyword> somewhere

So that the model would take an input word from a person and generate a couplet containing it.

Text input: "shuttle"

Ross Goodwin's captions

(using LSTM generation and image recognition)

2016 link

TECH: LSTMs (older than GPT2/3)

Poetic "captions": generate poetry from images

code link & paper -

the code no longer runs

but it could be replicated with

current tools

Allison Parrish - other ML models

"Reconstructions" uses a VQAE model to generate "chiasmus"

"Compasses", words generated in the phonetic space between 4 real words

See also GAC Nonsense Laboratory

Poety generated via image recognition - words provided for search / generation

TECH: Image recognition algorithms

word2vec / embeddings

 & other similarity methods

An embedding results in giving you words (or sentences, or images)  that are "similar" in some way, such as context of usage

Text

Color represents a normalized distance between the original word (random "nouny"*-words) and its next closest relative in the embedding model.

 

Blue: Closer, Pink: Further Away. (So, options for "sound" are going to be more different.)

word2vec model:

pre-configured poms:

Click on a colored word like "sound"...  pick a replacement from it's nearest neighbors in the model....

and you edited the poem :)

Original haiku by Bashō:

With a different embedding model:

one edit:

tech: word2vec / embeddings

AI Poetry Research - More and More Languages and Forms

Chinese (Classical, Modern, Lyrics...), Urdu and Hindi, Finnish, Japanese, ....  (see my page here

"AfriKI: Machine-in-the-Loop Afrikaans Poetry Generation" - trained model on fiction by South African novelist Etienne van Heerden in Afrikaans to generate poetry (LSTM)

A person chooses and orders the lines generated, in a co-collaboration. Like the word2vec edits.

Sample online AI poetry generators

Other Digital Poetry Experiments

Based on data, interactions, etc.

Open Street Map Haiku (not AI) by Satellite Studio - based on data in the map location

Every Thing Every Time - Naho Matsudo (Manchester, Newcastle..)

"Watch the live story of Newcastle unfold with an installation that turns city smart data into a constantly changing poem outside the Theatre Royal."

the web site is dead now, which makes me very sad.

Digitally remixed poetry, depends on the weather report (multi-award winning plus book)

Color Name Poems

 

by me

Ranjit Bhagnar's Pentametron, a twitter bot that got a lot of press in 2012+ for creating iambic pentameter poetry composed of found tweets.

E.g,

Gawker article

another article

Amelia was just fourteen and out of the orphan asylum;
        at her first job—in the bindery, and yes sir, yes
        ma'am, oh, so anxious to please.
     She stood at the table, her blond hair hanging about
        her shoulders, "knocking up" for Mary and Sadie,
        the stichers ("knocking up" is counting books and
        stacking them in piles to be taken away).

Blackout Poetry

Lots of tools/libs online to make it, e.g.

Pre-AI Inspirations: Oulipo

One Hundred Thousand Billion Poems by Raymond Queneau

The book is a set of ten sonnets printed on card with each line on a separate strip. As all ten sonnets have not just the same rhyme scheme but the same rhyme sounds, any lines from a sonnet can be combined with any from the nine others, allowing for 1014 (= 100,000,000,000,000) different poems

(could be made with code easily)

Choose Your Own Adventure Poetry

Autocorrect poetry

...

Google "search" poems

since 2013... or earlier (link)

"Search" poems

Search for rhyming lines in a

corpus of poems / text

TECH: RiTa.js

"Search" by keyword & generation

input a word, get a couplet containing it, and build up a poem doing this.

Text

(my work)

TECH: ElasticSearch db

A Few Refs: Digital Poets

 

 

 

 

Multimodal AI - Images and Text Models

text2image and image2text, primarily

Text2Image Tools

Given a text "prompt", generate an image. You can now do this in Midjourney and at OpenAI's DALL-E for payment -- or use an online code notebook (Colab) that takes a long time to run.

OpenAI's

DALLE-2

Midjourney

Try yourself, in open beta now.

demo it.

Artists Are Using Text2Image Tools

 

  • Comics / graphic novels
  • Stories
  • Animation
  • Textures - Blender guy
  • NFTs
  • Character design / concept design for games, movies

Artstation

Other Tools for Text2Image, Less Sophisticated

Multimodal artistic Search: Writing With Open Access - by Jono Brandel

search maybe uses the CLIP algorithm in text2image models

Image2Text

  • Captioning
  • Question Answering
  • Content / Poetry generation
  • ....

A "novel" constructed of image caption/descriptions in a video

Story gen from Images

"MAGIC" Demo site

Telephone games

1. Starting prompt: "A lonely traveler walked by a lake"

2. Generate image using text with a code tool

3. Caption by an image2text app

(Goto 2)

 

4. Write out the story.

How it starts

my input text is on 0; the top caption is the prompt for 1

A grid view of the prompt used to generate each image, and the image. Each prompt is the top caption of the previous image. In a perfect AI model world, there would be no drift at all in contents/images.

How it ends...

"Frankenstein's Telephone" by J. Rosenbaum

The caption spawns an image, which is classified and divided into colored segments depending on what the neural network sees, another image is created using those abstract sections as a guide and a final caption is generated.

visualizing transgender and non-binary people

AttnGAN

2017

Midjourney

now

Dalle-2

now

thread link

Joy Buolamwini: "AI, Ain't I A Woman?" (2018)

Machine translation

a very few examples, to stimulate some ideas?

"untranslatable"

Selections from IG poetry poster fossilisedflowers

Variation piece based on 47 translations into English of Dante’s opening lines of The Inferno.

English

Hi

"hi"

French

Salut

"hi"

Korean

etc.

안녕

Telephone Translation (mine)

listen: there is a good universe beside her; let's go
listen: there is a good universe nearby; go
listen: there's a damn good universe; gone
Listen: Next door is a very good corner. So let's go
Listen: The side door is a very good angle. Let's go
listen: there is a damn good universe; go
Please try listening. There is a very good universe
around. go
Please try to listen. There is a very nice corner on the
side. Okay, let's go
Listen: The side door is a very good corner. Let's go
Listen: there is a very good environment. Let's go
Listen: The neighbor has a very good angle. So let's go
English-Kurdish
English-Albanian
Kurdish-Finnish
Belarusian-German
Kurdish-Finnish
German-Albanian
Japanese-Albanian

Japanese-Kurdish

Somali-Kurdish
Haitian Cr.-Kurdish
German-Finnish

ee cummings   "listen: there's a hell of a good universe next door; let's go"

universe → corner | corner → angle | next door → side door or neighbor | let's go → go...

Tech: Translation APIs

Other NLP applications of possible interest

Dialogue Agents

"Sigana: Tales of Lawino"

telling immersive VR folk stories with multimodal AI conversational agents

(work by Melisa Allela)

Multimodal VR Storytelling With Chatbots

Sentiment Analysis

"The winding geometry’s angle and its base ascent are respectively driven by sampling a sentence from the book and by performing a sentiment analysis on it."

"Stendhal", by Damien Seguin

Is it positive, negative, neutral; happy, sad, excited, etc....

Style Transfer of Text

See also, Exercises in Style, Raymond Queneau

Information Extraction

You might then use the facts in a new form, like a game!

Networks of relations in texts

and Text Art

See my links on text vis & art

A Few Afro-centric NLP Links - See Jade's Talk!

Sigana, Tales from Lawana by Melissa  Link

Thanks... and I'm here to help

If you have tech stack questions, I can point you to tools or methods!

Tech Notes on AI Text Gen Models

  • "State of the Art" for long text generation are large models like GPT-3 (proprietary), GPT-J (open-source), NEOX...
    • In these models, you give a text prompt, and it "completes" it.
    • They are hard or expensive to "tune" on different texts; usually they are "prompt-engineered" with examples to steer them.
    • Awesome GPT-3 is a list of apps built on GPT-3
  • Same class, but smaller, and more affordable: GPT-2. Can be fine-tuned and hosted for apps.  Not as high quality.
  • If you just want single words, BERT/T5 etc. can be used. Those will give you missing words in a sentence.

Artistic NLP Projects

By Lynn Cherny

Artistic NLP Projects

  • 1,099