Text Analysis Without Programming

Lynn Cherny

Visiting Knight Chair

School of Comm, U of Miami

@arnicas / lynn@cherny.com

or click right/down arrow

to go into a topic.

use the space bar to hit every slide!

Caveat: A talk at the intersection of data vis, journalism, and digital humanities

(Also: Focus here is on free or very cheap tools)

What is text data?

Unstructured text write-ins on surveys
PDF Documents
Books (free or not)
Articles (magazine, news, blog...)
Comments on articles
Tweets, Facebook posts
Reviews (books, restaurants)
Prisoners' last words on Death Row
TV show closed captions
Debate or speech transcripts
Bills in Congress
Email archives
Recipes
Student essays...

Some Levels of Analysis

Word level: What words are used?
Word in context: Where are words used?
N-grams or phrases: What combinations of words are used?
Structure of the text:
- Sentence length
- Genre attributes
Relationships between texts
- Similarity/difference
- Metadata -- citation networks
Historical trends in corpora

PDFs are a sad text data reality

WSJ Website with full text search

Data.gov

PDF to Text

Command line utils for Linux: Poppler utils
Online and Windows software (free): Bunch of Links/Reviews
Mac software: One line, anyway
Batch convert on Mac/Windows with scripting (counts as a little bit of code, definitely knowledge of command line)

Demo one online

Some Document Corpus Tools Will Take and Convert PDFs for you

E.g., Overview Project

Document Cloud

But beware... a lot of tools only take text, so you have to do the work first.

Handling Hillary Clinton Emails on a Mac...

Footer on WSJ site: "Download zip files of each document release: May (Benghazi committee), June, July, August and September."
I got September.
Install xpdf from this site
At bash prompt: 'for f in *.pdf; do pdftotext $f; done'
Move .txt files (and .pdf) into their own directories.

Other data sets I'll use today...

Transcripts of the most recent Democratic Candidate and Republican candidate debates (scraped from WaPo)
Last statements from prisoners executed in Texas

Simple Word Counts

WSJ (Oct 14)

Using "grep" and "wc"

> grep CLINTON dem_debate_2015_10_13_wapo.txt 

CLINTON: Well, thank you, and thanks to everyone for hosting this first of the Democratic debates.

CLINTON: Well, actually, I have been very consistent. Over the course of my entire life, I have always fought for the same values and principles, but, like most human beings -- including those of us who run for office -- I do absorb new information. I do look at what's happening in the world.

CLINTON: No. I think that, like most people that I know, I have a range of views, but they are rooted in my values and my experience. And I don't take a back seat to anyone when it comes to progressive experience and progressive commitment.

CLINTON: I'm a progressive. But I'm a progressive who likes to get things done. And I know...

.... many more lines....

>grep CLINTON dem_debate_2015_10_13_wapo.txt | wc -l

      74

>grep SANDERS dem_debate_2015_10_13_wapo.txt | wc -l

      70

>grep WEBB dem_debate_2015_10_13_wapo.txt | wc -l

      35

Command line!

> wc -w dem_debate_2015_10_13_wapo.txt 

22953 dem_debate_2015_10_13_wapo.tx

> wc -w gop_debate_2015_9_16_wapo.txt 

35127 gop_debate_2015_9_16_wapo.txt

wc is unix for "word count." wc -w is just count words. You can also wc -l (lines).

Also see Word Count in Word 2013 How-To

Word Counts in Context :)

"Last Night's Debate Was Longer than The Book of Genesis" (C Ingraham, WaPo)

On Hillary's emails

>wc -w * | sort

.... lots of them go by...

1800 C05781926.txt  
1897 C05782687.txt  
2202 C05785187.txt  
2562 C05782645.txt  
2705 C05782303.txt  
3879 C05782890.txt  
4266 C05782607.txt  
4322 C05782571.txt
5697 C05781825.txt
211567 total

Screenshot of shame - trying to change file format on a Mac with regex...

I'm not going to go through this with you, sometimes programming is easier.

Concordance Tools

E.g., AntConc suite (free, all platforms)

Site

Concordances and Basic Corpus Concepts

word counts (how many times each word appears)
keyword in context (KWIC)
collocations (words occurring with a term)
n-grams (sequences of N words)
stop words (words that are common and may be filtered out from analysis)
sometimes, parts of speech (noun, verb, etc)

Keyword in Context Views

Plots

Using AntConc, at what points in the transcripts of the last GOP candidate debate and Democratic candidate debate did "(APPLAUSE)" occur?

transcript source scraped off Washington Post, using AntConc

Look familiar?

WaPo

N-Grams, or word groups that occur together

Collocates for word "gun"

AntConc on debate transcripts

Network Collocations in Overview Project

Overview Project on sample of Clinton emails

Context of the N-Grams

Word Trees

Formerly in the free Many Eyes, now requires code in Google Charts

Web-Based: Voyant

Voyant

Web Based: Lexos is Great.

Lexos link (does a lot more too)

Edging into Stylistics / Genre....

Site

Word Clouds

Wordle site -- uses Java applet, only runs in Firefox/Safari for me

A few Issues with Word Clouds

People don't know what they indicate (see research paper by Viegas, Wattenberg, Feinberg)
Inadequate screening of "stop words" -- filter out common "obvious" words before making them
Color/layout choices can also influence perception

But geez do people love them.