On this page
Our core corpus is 403 texts from the Temple University SCRC Paskow Science Fiction Collection (opens in a new tab). It is mostly New Wave SF novels published between 1945 and 1990, along with short story anthologies and magazines. The texts were scanned and converted to .txt files with ABBYY FineReader. After post-processing and cleaning the OCR output, we curated several datasets at different levels of granularity.
Extracted features#
Most texts in the corpus are still in copyright, so we cannot share the full works. We can share extracted features: disaggregated texts, word frequency counts, and syntactic and semantic descriptors. HathiTrust pioneered this approach with its Extracted Features dataset (opens in a new tab) for millions of volumes. Extracted features still support many kinds of analysis, including topic modeling. These datasets are free for public use:
Full texts
EF_Full_ Texts Disaggregated ("bag of words") versions of all 403 texts in the corpus, with cleaned and uncleaned variants and basic metadata (title, author, year, word count).
Chapters and chunks
EF_Chapters_ and_ Chunks All 403 texts split into segments of up to 1,000 words, each disaggregated into a bag of words. Uniform segments suit topic modeling.
Chapters only
EF_Chapters_ Only The 306 texts whose chapters could be detected from OCR tags, split by chapter and disaggregated into bags of words.
Named entities
EF_Named_ Entities BookNLP entity files listing the people, organizations, locations, vehicles, and other named entities in each text, with part of speech and position.
Supersense tags
EF_Supersense_ Tags BookNLP supersense files tagging a broader range of tokens, including emotions, substances, artifacts, animals, and cognition, with part of speech and position.
The three bag-of-words datasets each contain two versions of every text. One is “cleaned” (lowercased, with punctuation, extra spaces, and numbers removed) and one is not. They also include basic metadata about each book: title, author, publication year, and word count. Metadata curation is ongoing, and we plan to add publication details, descriptions, and Library of Congress genre categories.
Loaded into Python, the head of a dataframe looks like this:
Loading the data#
All of our extracted-features sets are in the SF-Corpus organization on Hugging Face (opens in a new tab). You can download them from the website, like files on GitHub, or load them straight into Python with the datasets library. The library downloads and caches them locally:
# pip install datasets
from datasets import load_dataset
# Load the chapters-and-chunks version of the SF Nexus corpus
dataset = load_dataset("SF-Corpus/EF_Chapters_and_Chunks")
# Work with it as a pandas DataFrame
df = dataset["train"].to_pandas()
For more on working with Hugging Face datasets, see the loading guide (opens in a new tab).
How to cite#
If you use these datasets, please cite the SF Nexus and the specific dataset, for example:
SF Nexus, Temple University Libraries. EF_Chapters_and_Chunks [dataset]. Hugging Face. https://huggingface.co/datasets/SF-Corpus/EF_Chapters_and_Chunks (opens in a new tab).
For the project’s approach, cite Wermer-Colan and Kopaczewski, “The New Wave of Digital Collections” (opens in a new tab), Transactions of the American Philosophical Society 110, no. 3 (2022).
How the datasets were made#
We extracted features from the collection with several Python and R pipelines. They are available as Jupyter and Google Colab notebooks in our extracted-features-notebooks repository (opens in a new tab). Jeff Antsen discusses the method at length on the Scholars Studio blog in “Curating Copyrighted Corpora: An R Script for Extracting and Structuring Textual Features” (opens in a new tab) (2019).
Pipeline 1: Text sectioning and disaggregation#
Our first pipeline sections and disaggregates the science fiction texts. Breaking long texts into sections (here, chapters and n-word chunks) helps researchers working with large texts and corpora. Many books in the collection run to 50,000 words or more. Splitting them into chapters and chunks of 1,000 words or fewer speeds up analyses like topic modeling and Word2Vec. It also lets researchers ask more granular questions: for example, which topics or themes emerge across a book’s chapters, and what they suggest about its narrative arc.
Disaggregation turns full texts into “bag-of-words” models that humans cannot read. These can be shared where copyrighted full texts cannot, and they still work for methods like LDA topic modeling that depend mainly on word frequencies rather than word order.
Input: machine-readable text files of 403 science fiction books.
To build an extracted-features set of the full corpus, we first split each text into 1,000-word segments. These uniform chunks suit analyses like topic modeling, whereas chapters (below) vary in length. Here is the approach in Python (the R pipeline is documented in the repository README (opens in a new tab)):
# Define chunking function
def split(list_a, chunk_size):
for i in range(0, len(list_a), chunk_size):
yield list_a[i:i + chunk_size]
# Set desired size of chunks
chunk_size = 1000
# Create new list for chunked sentences
chunked_ch = []
# Perform chunking function on each row of tokens
s = new_chapters_df['Tokens']
for content in s:
chunks = list(split(content, chunk_size))
# Add to new list
chunked_ch.append(chunks)
After segmenting, we sorted the words in each segment alphabetically. This turns human-readable text into bags of words.
Output 1: disaggregated versions of 403 texts, split into 1,000-word segments (opens in a new tab).
We also split texts by chapter rather than by 1,000-word chunk. This was possible for a large part of the corpus (306 texts) because chapters were tagged consistently during OCR cleanup. Python code retrieves all text between each book’s START OF BOOK and END OF BOOK tags, leaving out title pages, dedications, acknowledgements, and so on. It then splits the text at every occurrence of CHAPTER:
# Count number of chapters in each text
chapter_counts = books_cleaned['Text'].str.count('CHAPTER')
# Append chapter counts to dataframe
books_cleaned["Chapters"] = chapter_counts
# Make new cell each time new chapter starts
new = books_cleaned["Text"].str.split("CHAPTER", expand = True).set_index(books_cleaned['Title'])
The resulting DataFrame holds each chapter of each book as a separate text:
As with the chunks, we then sorted the words in each chapter alphabetically to turn them into bags of words.
Output 2: disaggregated versions of 306 texts, split into chapters (opens in a new tab).
Finally, we disaggregated each full text for researchers who don’t need smaller units.
Output 3: disaggregated versions of 403 full texts (opens in a new tab).
Pipeline 2: Part-of-speech tagging, named entities, and supersenses#
Our second pipeline “enriches” the corpus by extracting parts of speech, named entities, and supersense tags (such as “animal,” “artifact,” “body,” and “cognition”) from each text. BookNLP retrieves all of this and more. It is a natural language processing pipeline for book-length texts, developed by David Bamman at UC Berkeley. Its functions include:
- tokenization
- sentence boundary detection
- part-of-speech tagging
- dependency parsing
- entity recognition
- entity mapping (aligning aliases to a single character ID)
- gender prediction
Digital humanists have used BookNLP to study character (opens in a new tab) and gender bias in children’s literature (opens in a new tab). For a fuller introduction, see Introduction to BookNLP by W. J. B. Mattingly (opens in a new tab).
Input: machine-readable text files of 403 science fiction books.
We ran the full (aggregated) texts through BookNLP in Python. It takes only a few lines:
# Define folder where files are stored to run BookNLP on
files = [f for f in os.listdir(path) if os.path.isfile(f)]
# Loop to run BookNLP on each file
for f in files:
# Define filename as each file in folder
inputFile = f
# Set folder for BookNLP results
outputDir = "BookNLP_Results/"
# Set ID as name of each file
idd = f
# Run BookNLP on each file
booknlp.process(inputFile, outputDir, idd)
This produces a folder of BookNLP files for each text. Two kinds of files are published on our Hugging Face page.
Output 1: entity files for 403 science fiction books (opens in a new tab). These list the named entities in each text (people, organizations, locations, vehicles, and more), with each entity’s part of speech and position in the text.
Output 2: supersense files for 403 science fiction books (opens in a new tab). These tag a broader range of tokens beyond people, places, and locations, including tokens identified as “emotions,” “substances,” or “artifacts.” Each tag also records part of speech and position in the text.
For a fuller overview of BookNLP’s output, see Chapter 3, “The Output Files” (opens in a new tab) of Mattingly’s Introduction to BookNLP.
Full-text analysis#
Researchers can analyze the full texts, without reading them, in two ways. On site at Temple, the Scholars Studio provides supervised access for non-consumptive research (see OCR). Remotely, researchers have been able to analyze HathiTrust volumes, including in-copyright works, in the HathiTrust Research Center (HTRC) Data Capsule, a secure computing environment. Temple’s data capsule appliance (opens in a new tab) is a way to set up and run that kind of environment locally.
Take-down notice#
This site shares only extracted features of copyrighted fiction. No copyrighted work is made available to read. These derived files are provided for education and research. Temple University Libraries has credited rights holders where possible. If you hold rights to materials in our digitized collections that are not credited, please let us know so we can keep our information accurate.
If you are a rights holder and believe this site includes material you have not permitted (or that is not covered by an exception under U.S. copyright law), you can request its removal by writing to digitalscholarship@temple.edu. For more on non-consumptive research, see HathiTrust’s Non-Consumptive Use Research Policy (opens in a new tab).