Week 9: Automated Extraction of Retail Inventories (Web Scraping)
Programmatic extraction of live pricing data: DOM parsing, HTTP requests, and structural data hygiene.
Core reading: Automated Extraction of Retail Inventories: Legal and Technical Boundaries (Kim, Data Science Quarterly, 2021).
The Lecture#
To this point, our economic modeling has relied on manual, localized audits. However, manual calculation of the ABV-to-dollar index is fundamentally insufficient for macro-level optimization. Retail pricing is a highly dynamic apparatus; supermarkets rely on localized discount cycles, hidden clearance anomalies, and algorithmic price matching that can vanish within hours. To fully exploit these inefficiencies, we must transition to applied computer science and operate at scale.
This week addresses the programmatic extraction of live inventory data from the digital storefronts and APIs of major Australian liquor retailers. We will cover the specific technical methodologies required to parse Document Object Model (DOM) architectures, execute automated HTTP requests, and structure raw, unstructured HTML into clean, statistically valid CSV datasets. Crucially, we will delineate the strict technical boundaries of this extraction, distinguishing between legitimate academic data harvesting and malicious server disruption.
Applied Computational Lab#
In this week’s laboratory, you will abandon the manual calculator and initialize your Python environment. This lab serves as the direct technical prerequisite for your Data Scraping Exercise assessment.
Utilizing libraries such as BeautifulSoup4 and requests, you will draft a rudimentary scraping script targeting a provided, sanitized retail sandbox environment. Your objective is to programmatically extract raw HTML nodes, isolate the pricing, ABV, and volumetric strings, sanitize the variables (stripping currency symbols and extraneous whitespace), and append them to a structurally sound dataframe. You are strictly required to hardcode a crawl-delay variable into your while loops to ensure your automated requests respect standard server limits.
Mandatory reading#
-
The Billion Prices Project: using online prices for measurement and research — Cavallo & Rigobon, Journal of Economic Perspectives 30(2), 2016, 151–178. This paper demonstrates the macroeconomic validity of our technical methodology. Cavallo and Rigobon detail the process of scraping retail prices at a national scale to calculate inflation indices. You are required to read this specifically for their section on measurement biases. The errors they document—substitution bias, survival bias, and missing data—are the exact computational failure modes that will invalidate your dataset if your DOM parsing logic is flawed.
-
Web scraping for research: legal, ethical, institutional, and scientific considerations — Brown, Gruen, Maldoff, Messing, Sanderson & Zimmer, Big Data & Society 12(4), 2025. The academic integrity conduct rules outlined on the SLOP4230 course policies page are simply the abbreviated, administrative version of this paper. This document outlines the absolute legal and ethical parameters of automated data extraction in a university context. You are expected to read and understand these institutional considerations before you execute your first live server request, not after your IP address has been blacklisted.
Simulated scraper output
Illustrative, not live — the figures scrolling below are the same worked catalogue as the toolkit, run through the same standard-drink and index formulas. Week 9's lab builds the version that requests a real page.
View the same output as text
$ python scrape.py --retailer demo-liquor.test --min-skus 9 # crawl-delay: 2.0s per request (see /policies/) GET https://demo-liquor.test/catalogue/cask-4l -> 200 OK Cask white, 4 L | 4000mL | 10.5% ABV | $14.50 -> 33.1 std drinks, index 2.285 GET https://demo-liquor.test/catalogue/cask-2l -> 200 OK Cask red, 2 L | 2000mL | 11% ABV | $9.00 -> 17.4 std drinks, index 1.929 GET https://demo-liquor.test/catalogue/btl-wine -> 200 OK Bottled shiraz, 750 mL | 750mL | 13.5% ABV | $11.00 -> 8.0 std drinks, index 0.726 GET https://demo-liquor.test/catalogue/beer-30 -> 200 OK Full-strength block, 30 × 375 mL | 11250mL | 4.8% ABV | $58.00 -> 42.6 std drinks, index 0.735 GET https://demo-liquor.test/catalogue/beer-6 -> 200 OK Craft six-pack, 6 × 330 mL | 1980mL | 5.2% ABV | $24.00 -> 8.1 std drinks, index 0.338 GET https://demo-liquor.test/catalogue/spirit-1l -> 200 OK Bottom-shelf vodka, 1 L | 1000mL | 37.5% ABV | $42.00 -> 29.6 std drinks, index 0.704 GET https://demo-liquor.test/catalogue/spirit-700 -> 200 OK Mid-shelf gin, 700 mL | 700mL | 40% ABV | $48.00 -> 22.1 std drinks, index 0.460 GET https://demo-liquor.test/catalogue/rtd-10 -> 200 OK RTD cans, 10 × 375 mL | 3750mL | 4.5% ABV | $46.00 -> 13.3 std drinks, index 0.289 GET https://demo-liquor.test/catalogue/rtd-4 -> 200 OK Premium RTD, 4 × 330 mL | 1320mL | 6% ABV | $22.00 -> 6.2 std drinks, index 0.284 Parsed 9 SKUs, 0 failures. Sorted by index, descending.
The week 9 lab: First hundred rows
Before the lab#
Bring a laptop and your chosen retailer’s online liquor catalogue. Bring whatever scraping code you already have, working or not — a broken scraper is a better starting point for this lab than an empty file.
In the lab#
Conduct before code. Check robots.txt first, set a crawl delay of at
least two seconds, and identify your client honestly. This is a course
rule stated on the policies page, and it is also the
difference between a method another researcher could repeat and one that
gets an IP range blocked for the whole cohort.
The four failure modes. The week’s reading catalogues them, and all four will appear in your first hundred rows:
- volume in the title, not a field — “Cab Sav 750ml” parses; “Cab Sav 75cl” and “Cab Sav 6x330” do not, unless you make them
- ABV missing entirely — common on beer and near-universal on multipacks; inferring from category is allowed, recording that you inferred it is required
- promotional price in the price field — the number changes under you between runs, so timestamp every row
- the same product listed twice at different sizes, which double-counts unless you key on something better than name
The honest column. Add a column recording, per row, whether ABV was stated or inferred. The assessment’s reproducibility criterion is substantially about whether a reader can tell your measured data from your assumed data, and a column is the cheapest way to let them.
Afterwards#
You leave with a running scraper, a hundred timestamped rows, and a count of how many of them you had to guess at. The Data Scraping Exercise is due the week after this one.
What you leave the lab able to do
Written so a reader can tell whether it has been met without asking you. The assessments assume it has.
- your scraper returns at least 100 rows with name, volume, ABV and price populated
- you have a documented crawl delay and a check against the site's robots.txt
- you can name which of your rows have an inferred rather than a stated ABV, and how many
Teaching team
- Marisol Quaye — convenor