Gathering the Data
Identify Sources
Challenge
Independent militaria dealers describe and organize similar objects in different ways. Product titles, prices, availability labels and image layouts need source-specific handling.
Approach
I focus on specialist dealer websites and maintain configuration for their product pages. My collecting and antiques background helps me interpret the source material and choose useful fields.
Current scope
Milivault’s public homepage displayed 147 dealers and 531,639 tracked listings on October 13, 2026. The catalog includes archived and unavailable listings; these counters are a dated snapshot, rather than a measure of unique objects or guaranteed freshness.
Once I had the right data sources, the next step was to design a scraper that could handle their diversity.
(Photo: One of my favorite shops before it closed — exactly the kind of independent source this project was built on.)
Build the Scraper
Challenge
Each dealer has different HTML, product identifiers, price formats and availability signals. A reusable scraper still needs source-specific rules and ongoing maintenance.
My work
I build and maintain Python collection scripts using BeautifulSoup, with Selenium where browser rendering is needed. Dealer JSON profiles hold selectors and parsing rules. The workflow checks listing pages, identifies new products and updates existing records.
I use product URLs and dealer SKUs as the main identity signals. Titles, descriptions and images can support comparison, but titles and descriptions are less dependable identifiers.
Storage and maintenance
New product fields are extracted, cleaned and stored in PostgreSQL on AWS RDS, with images in S3. AWS EC2 runs collection work. I use Codex extensively alongside manual configuration, testing and troubleshooting.
Improving the Workflow
The diagram documents the move from repeatedly checking individual stored URLs toward checking dealer listing pages for new and changed records.
Existing products receive a last-seen timestamp when observed. New products receive detail extraction and cleaning. Availability changes require valid source evidence: a failed request or broken selector is not proof that an item sold.
If a run looks wrong, I check the dealer’s website first, then the JSON profile and selectors. I preserve existing availability while diagnosing the problem.
The aim is to reduce unnecessary requests and processing while maintaining useful inventory history. The diagram illustrates the approach; it does not establish a measured speedup or universal crawl schedule.
System at a Glance
Read the source: Python reads the dealer’s product listing pages using site-specific selectors.
Match records: Product URLs and dealer SKUs help distinguish existing records from new listings.
Track observations: Refresh the last-seen timestamp when a product is observed and process supported availability changes.
Extract details: Collect fields such as title, description, price and images for new products.
Clean and store: Normalize the extracted fields, store structured records in AWS RDS PostgreSQL and images in S3.
Investigate failures: Check source pages and dealer configuration before changing stored availability.
Balancing Freshness & Efficiency
Challenge
Repeatedly downloading unchanged products creates unnecessary work for Milivault and its source websites.
Approach
I separate checks for new or changed listings from full detail extraction. Source-specific behavior, valid observations and last-seen timestamps guide the update process.
Ongoing improvement
My October 2026 update describes work to avoid repeat requests and unnecessary AI processing. Freshness depends on the source and successful runs; I do not claim that every dealer is checked on one fixed schedule.
Historical exploration of when Bevo Militaria listings were first observed. Observed timestamps depend on collection schedules and do not necessarily equal a dealer’s publication time.
Adapting to Edge Cases
Handling Edge Cases
Some dealers require browser rendering, while others change their HTML or product status labels. I maintain source-specific configuration and investigate changes when extraction fails.
My troubleshooting sequence is to inspect the live website, compare its structure with the dealer JSON profile, and test the selectors. Stored availability stays unchanged until the source behavior is understood.
Selenium supports JavaScript-dependent pages. PostgreSQL records and stored observations support later analysis, but missing products and scrape failures need separate treatment.
Skills and Technologies
Data Collection & Processing
Python, BeautifulSoup, Selenium and pandas.
Dealer JSON profiles, field extraction and cleaning, product identity checks and last-seen observations.
Classification & AI Integration
scikit-learn experiments, text features and OpenAI API integration.
Classification proposals, source context and human review. Model confidence and processing coverage are distinct from evaluated accuracy.
Web Development
Django powers the public Milivault site and is the current direction for internal classification review.
Earlier Streamlit tools are historical prototypes. Local or private features are identified separately from public releases.
Cloud Infrastructure
AWS EC2 for collection work, AWS RDS PostgreSQL for structured records and AWS S3 for images.
The Django site uses Render with AWS-backed services.
Database Work
PostgreSQL and SQL for product records, classification fields, updates and operational investigation.
Record identity, source-specific data and timestamps are central to the collection workflow.
Development & Deployment
Git and GitHub for tracking changes, with deployment and operational work across the Django site and AWS services.
I own the project end to end, using Codex extensively alongside manual work and verification.
