Exploring Lewis Grassic Gibbon First Editions¶

Created August-September 2020 for the National Library of Scotland's Data Foundry by Lucy Havens, Digital Library Research Intern

About the Lewis Grassic Gibbon First Editions Dataset¶

Lewis Grassic Gibbon is an early 20th century Scottish novelist who also published under his birth name, James Leslie Mitchell. He was an prolific writer for the short period of time (5 years) that he published fiction and non-fiction, and the NLS collection contains first editions of all his published books. Gibbon's stories often featured strong central female characters, unusual for an early 20th century writer. Gibbon's literary influence continues to be felt today: his book A Sunset Song was voted Scotland's favorite novel in 2016, and contemporary Scottish writers such as Ali Smith and E.L. Kennedy have noted Gibbon's influence on their own writing.

• Data format: digitised text
• Data creation process: Optical Character Recognition (OCR)
• Data source: https://data.nls.uk/data/digitised-collections/lewis-grassic-gibbon-first-editions/

Table of Contents¶

1. Preparation
2. Data Cleaning and Standardisation
3. Summary Statistics
4. Exploratory Analysis

Citations¶

• Alex, Beatrice and Llewellyn, Clare. (2020) Library Carpentry: Text & Data Mining. Centre for Data, Culture & Society, University of Edinburgh. http://librarycarpentry.org/lc-tdm/.
• Bird, Steven and Klein, Ewan and Loper, Edward. (2019) Natural Language Processing with Python – Analyzing Text with the Natural Language Toolkit. O'Reilly Media. 978-0-596-51649-9. https://www.nltk.org/book/.

0. Preparation¶

Import libraries to use for cleaning, summarising and exploring the data:

To explore the text in the Lewis Grassic Gibbon First Editions collection, we'll mainly use the Natural Language Toolkit (NLTK), a library written for the programming language Python.

The nls-text-gibbon folder (downloadable as Just the text data from the website at the top of this notebook) contains TXT files of digitised text, with numerical names, as well as a CSV inventory file and a TXT ReadMe file. Load only the TXT files of digitised text and tokenise the text (which splits the text into a list of its individual words and punctuation in the order they appear):

Note: If you'd like to see how to specify a single TXT file to load as data, check out the Jupyter Notebook for the Britain and UK Handbooks!

It's hard to get a sense of how accurately the text has been digitised from this list of 15 tokens, so let's look at one of these words in context. To see phrases in which "Scots" is used, we can use the concordance() method:

There are some mistakes but not too many!

Note how NLTK's concordance method works: the word "Scots" appears with different meanings, sometimes referring to the language, other times referring to the people. NLTK has a tagging method that identifies the parts of speech in sentences, so if we wanted to focus on the language Scots, we could look for instances of Scots being used as a noun. If we wanted to focus on the people Scots, we could look for instances of Scots being used as an adjective. This method wouldn't return perfect results, though. For example, we could improve our results by checking for instances of "Scots" being used as an adjective before the word "dialects," for example.

We'll wait to dive into this sort of text analysis until a bit later, though!

0.1 Dataset Size¶

First, let's get a sense of how much data (in this case, text) we have in the Lewis Grassic Gibbon First Editions (LGG) dataset:

Note that I've print Tokens rather than words, though the NLTK method used to count those was .words(). This is because words in NLTK include punctuation marks and digits, in addition to alphabetic words.

0.2 Identifying Subsets of the Data¶

Next, we'll create two subsets of the data, one for each journal. To do so we first need to load the inventory (CSV file) that lists which file name corresponds with which journal. When you open the CSV file in Microsoft Excel or a text editor, you can see that there are no column names. The Python library Pandas, which reads CSV files, calls these column names the header. When we use Pandas to read the inventory, we'll create our own header by specifying that the CSV file as None and providing a list of column names.

When Pandas (abbreviated pd when we loaded libraries in the first cell of this notebook) reads a CSV file, it creates a table called a dataframe from that data. Let's see what the Gibbon inventory dataframe looks like:

Since we only have 16 files, we returned the entire dataframe above. With larger dataframes you may wish to use df.head() or df.tail() to print only the first 5 or last 5 rows of your CSV file (both of which will include the column names in the dataframe's header).

Now that we created a dataframe, if we want to determine the title of a Gibbon work based on it's file ID, we can use the following code:

NLTK stores the lists of tokens in the corpus_tokens variable we created by the file IDs, so it's useful to be able to match the file IDs with their book titles!

1. Data Cleaning and Standardisation¶

1.1 Tokenisation¶

Variables that store the word tokens and sentence tokens in our dataset will be useful for future analysis. Let's create those now:

For some types of analysis, such as identifying people and places named in Gibbon's works, maintaining the original capitalization is important. For other types of analysis, such as analysing the vocabulary of Gibbon's works, standardising words by making them lowercase is important. Let's create a lowercase list of words in the LGG dataset:

Perfect!

1.2 Reducing to Root Forms¶

In addition to tokenisation, stemming is a method of standardising, or "normalising," text. Stemming reduces words to their root form by removing suffixes. For example, the word "troubling" has a stem of "troubl." NLTK has two types of stemmers that use different algorithms to determine what the root of a word is.

The stemming algorithms below can take several minutes to run, so two are provided below with one commented out (the lines begin with #) so it won't run. If you'd like to see how the stemming algorithms differ, uncomment the lines by highlighting them and pressing cmd + /.

First, though, let's see what stems of LGG data look like with the Porter Stemmer:

Another approach to reducing words to their root is to lemmatise tokens. NLTK's WordNet Lemmatizer reduces a token to its root only if the reduction of the token results in a word that's recognised as an English word in WordNet. Here's what that looks like:

2. Summary Statistics¶

2.1 Frequencies and Sizes¶

Now that we've created some different cuts of the LGG dataset, let's start investigating the frequency of terms as they appear across the dataset. One way to do so is with a frequency distribution, which is a line chart that shows how many times a token appears in the dataset. Let's plot the frequency distribution of alphabetic words in the entire LGG dataset, excluding stop words (for example: a, an, the).

Step 1: First, let's create a new list of lowercase words in the LGG dataset that excludes stop words and calculate it's frequency distribution:

Notice the â that appears as the most common token on its own, but also at the end of other common tokens. This is probably an OCR error (a mistake in the digitisation process). Let's see what other words appear near â to get a sense of its context:

Hmmm. â seems to appear in many different ways, so we probably shouldn't remove it entirely. From the sample of text displayed above, it looks as though sometimes it appears in place of ' or -, while other times it seems nothing should be there at all. It's not possible to know without comparing each occurrence of â to the original text manually.

Step 2: Since manual OCR correction is quite a time-consuming effort, for the purposes of a frequency distribution, we'll remove the âs for now:

Step 3: Now, we recalculate the frequency distribution and visualise it using NLTK and Matplotlib:

Next, we can look at multiple words at a time. Using collocations(), we can see which pairs of words occur together most often across the LGG dataset:

thousand years is a common bigram, or word pair. I wonder what other words appear in similar contexts to the word year...

Let's pick a single work of Gibbon's to and see what its collocations are:

Many of the most common bigrams are names!

2.2 Uniqueness and Variety¶

To measure the uniqueness and variety of words in Gibbon's works, we can calculate the lexical diversity of files in the LGG dataset. Lexical diversity measures the diversity of word choice, calculated by dividing the number of unique words by the total number of words in a work. Instead of simply counting the number of unique words in each of Gibbon's works, we divide by the total number of words in a work to normalise the metric, so we can compare the diversity of word choice of works that are of different lengths. Let's calculate the lexical diversity for each file in the LGG dataset and add this metric to our inventory DataFrame:

To make it easier to compare the lexical diversity scores, let's visualise them!

This makes it clear that Hanno, or, The future of exploration is by far the most lexically diverse in the collection. Interestingly, this was the first book he wrote (it was published in 1928)!

3. Exploratory Analysis¶

3.1 How does Gibbon's word choice evolve throughout his publishing life?¶

Using information from the digital.nls.uk website, we can quickly find and add the publication dates to Gibbon's works. Since this LGG dataset is first editions, we know that the publication dates of Gibbon's books at the NLS will be the initial, original dates.

Now let's create another visualisation of the lexical diversity of Gibbon's works, but this time we'll sort the data by chronologically by published date:

Since we don't know the months in which Gibbon's works were published, let's calculate Gibbon's yearly lexical diversity to get a better sense of the trend over time.

Step 1: First, we need to group Gibbon's works by the year in which they were published:

Perfect!

Step 2: Now we'll calculate the lexical diversity across all works published each year from 1928 through 1934 (excluding 1929, since the LGG dataset doesn't include any works published that year):

So Gibbon's lexical diversity does decrease over time, excepting a small increase in the last year he published, 1934!