Reducing OpenAI Classification Costs with Local ML Coverage Analysis
A Tableau case study showing how I identified where Milivault depends on OpenAI for classification and where local ML improvements would have the highest cost impact.Milivault uses classification to organize militaria products by item type, nation, and conflict. Early on, I assumed my local machine learning models were doing more of the heavy lifting than they actually were. I knew the item type model was performing fairly well, but I thought nation and conflict classification were also contributing more than they were.
Once OpenAI API costs started adding up, I wanted to understand exactly where the money was going and where local ML could realistically reduce dependency on AI.
This started as part of a broader cost-control effort. I had already been looking at scraper and database efficiency by batching more operations, reducing unnecessary updates, and tuning individual site scraping schedules. The next question was whether classification could be made more efficient too.
The goal was simple: use OpenAI where it is genuinely useful, but avoid depending on it for classifications that a local model should be able to handle.
Tools used: Python, pandas, Tableau, PostgreSQL export data, OpenAI API billing exports, and Milivault taxonomy JSON files.
The Problem
OpenAI is extremely useful for messy militaria data. Many dealer listings have poor titles, limited descriptions, inconsistent category labels, or images that provide more context than the text. In those cases, AI still has a place.
The problem is that small API costs become meaningful at scale. Milivault collects hundreds of products per day, and even fractions of a cent per classification can add up month after month.
I needed a way to answer three questions:
1. Where is local ML already helping?
2. Where is OpenAI still carrying most of the classification workload?
3. Which classification area should I improve first to reduce AI dependency?
Data and Cleanup Method
The first issue was that raw labels were inconsistent. The same category could appear with different casing, wording, or formatting. Some AI-generated labels also did not perfectly follow the allowed taxonomy. That made the data harder to analyze because visually similar categories were not always the same value in the data.
To fix this, I cleaned the classification values into official taxonomy keys and created source fields showing whether each classification came from local ML, OpenAI, or remained unclassified.
For each classification area, I created cleaned fields for:
· item type
· nation
· conflict
Each one tracked:
· the source of the classification
· the raw value used
· the cleaned taxonomy key
· whether that cleaned value matched the official taxonomy
This turned the data into something Tableau could use reliably.
Item Type Analysis
Item type was the area where I had already put the most effort into local ML. I had manually labeled a large number of products because the item type categories were sometimes close together or overlapping. I wanted the model to understand the difference between categories like insignia, field gear, medals, documents, uniforms, and other militaria objects.
The item type charts showed that local ML was helping, but not as much as I expected. Some categories had much higher fallback rates than I wanted. Even in areas where the model was doing well, like insignia and patches, OpenAI was still handling a large number of products because the volume was so high.
The biggest OpenAI workloads were in categories such as insignia and patches, field gear, and other high-volume item types.
This was useful because it showed that a category can have decent ML coverage and still be a major cost target if the volume is large enough. Improving item type ML is not just about fixing the weakest categories. It is also about improving high-volume categories where even a small percentage improvement can save a lot of AI calls.
Some categories are harder to improve. Miscellaneous collectibles are naturally messy, and rare categories like crew-served weapons do not have enough labeled examples to train a strong model yet. For item type, the next improvement is straightforward: more manually confirmed training data.
Nation Analysis
Nation classification was much more dependent on OpenAI than I expected.
The overwhelming majority of nation labels were still being handled by AI. I knew the nation model had less training data than item type, but seeing that the dependency was close to total was a wake-up call.
The most important nations for Milivault are Germany, the United States, and the United Kingdom because they appear so often in militaria listings. Improving local ML for those high-volume nations would have the largest practical benefit.
Nation classification is difficult because many dealer listings do not provide much text. Some titles are extremely short, and descriptions are sometimes missing or vague. Other items do not fit neatly into one nation, such as pieces connected to Yugoslavia, the Ukrainian SSR, or other historically complicated cases.
Still, the benefit of improving nation ML is clear. If local ML can handle the most common nation classifications more confidently, Milivault can reduce a large amount of OpenAI dependency.
Conflict Analysis
Conflict classification had a similar issue to nation classification. Very little of the workload was being handled by local ML.
At first glance, conflict seems like it should be easier because the labels are broad: WW2, WW1, Cold War, Modern, and so on. But in practice, conflict classification is more complicated than it looks.
The biggest issue is that conflict is based on time period and historical context. For example, interwar Germany is not always easy to classify. Some dealers group 1933–1939 material with WW2 Germany, while others treat WW2 more strictly as 1939–1945. Those decisions matter because they affect how products appear in search, filtering, and analysis.
Most Milivault products are concentrated in WW2, followed by WW1. That may reflect the sites I scrape, the broader militaria market, or both.
The biggest risk with conflict classification is not a single wrong label. It is entire groups of products being misclassified and disappearing from the places users expect to find them. Because of that, conflict is the classification area I would revisit first. Before improving the model, I need to make sure the taxonomy rules are clear enough.
Cost and Scale Analysis
After looking at classification coverage, I connected the analysis to OpenAI API billing data.
Across the analyzed period, total OpenAI API cost was about $872.71. At first, that number does not sound extreme, especially when spread across hundreds of thousands of collected products. But that is not the right way to think about the cost.
The API cost is not really tied to every product collected. It is tied to the products and classification fields that actually needed OpenAI. Each product can have three classification fields: item type, nation, and conflict. A product might use ML for one field and AI for the other two.
That means the better unit of analysis is not just products collected. It is AI-handled classification fields.
This made the dashboard more useful. Instead of asking, “What did OpenAI cost per product?” I looked at how much classification work was still being handled by AI and how that changed over time.
The ML coverage trend showed that local ML handled roughly 20–30% of classification fields over time. That is useful, but it is not enough. If Milivault keeps growing and ML coverage stays flat, API costs will continue to scale with the product database.
The next target should be getting ML coverage closer to 50–75%, while keeping OpenAI as a fallback for low-confidence or poorly described products.
Key Findings
The biggest thing I learned is that small costs matter at scale. Fractions of pennies do not feel important when looking at one product, but they become important when classification is running across hundreds of thousands of records.
The second finding is that local ML was helping, but not enough. Having 20–30% of classification fields handled by ML saved money, but it also showed that most of the classification workload was still dependent on OpenAI.
The third finding is that not all classification areas should be treated equally. Item type ML is already contributing and can be improved with more labeled data. Nation and conflict are much more dependent on AI and need more structural work.
The most useful chart was the ML coverage trend. It showed that coverage was staying in the same general range instead of steadily improving. That made it clear that the system needed more deliberate model improvement, not just more time.
Next Steps
The next step is to build a cleaner validation loop.
I want to manually confirm products labeled by both ML and AI so the training data stays clean. I do not want AI hallucinations or weak ML predictions feeding back into the model and making the system worse.
The first model area I would revisit is conflict. Before training a better model, I need clearer rules for how the taxonomy should handle edge cases like interwar items, pre-war political material, and historically ambiguous periods.
After that, I would expand manually confirmed labels for the highest-volume nation and item type categories.
Success in the next version would mean raising ML coverage from roughly 20–30% to somewhere closer to 50–75%, while still using OpenAI as a fallback when the local model is not confident.
Why Automation Was Necessary
This flow shows how WW2 products branch across nation and item type classifications. The density of connections shows why manually classifying every product would not scale, and why Milivault needs a structured pipeline using user-confirmed labels, local ML, and OpenAI fallback.
Final Takeaway
This project turned OpenAI cost reduction into a prioritization problem.
Instead of guessing where to improve local ML, I looked at the data and worked backwards. I identified where OpenAI was doing the most classification work, where local ML was already helping, and where model improvements would likely have the highest return.
The broader lesson is simple: reduce AI dependency where local ML can handle the work, and reserve OpenAI for messy, low-confidence cases where it adds the most value.
Projects Powered by This Classification System
What Is Your WWII German No Decal Helmet Worth in 2026?
Uses cleaned classification data to compare similar products and estimate market value.
Milivault Product Database
Browse 500,000+ militaria products organized by item type, nation, conflict, and availability.