MLOps Implementation for a Grid Intelligence Startup
MLOps for Computer Vision: From Notebooks to a Production Pipeline
How Agmis took a grid intelligence startup's computer vision models out of Jupyter notebooks and onto a reproducible AWS SageMaker pipeline - with every training run recorded, traceable and repeatable.
A Grid Intelligence Pioneer Transforming Utility Infrastructure
A venture-backed grid intelligence startup with a distinctive position in utility infrastructure assessment. Their approach: autonomous AI systems mounted on fleet vehicles, scanning and evaluating power grid assets at scale. To date, their models have assessed more than 500,000 poles for major US utilities - pushing the boundaries of what computer vision can do for utility asset management.
Rapid Growth Exposed Cracks in ML Infrastructure
Inefficient Training Processes
Model development lived primarily in Jupyter notebooks - an approach lacking the structure reproducibility requires. A notebook holds state that exists in one place: environment, execution order, local data paths, an uncommitted preprocessing change. The cost shows up twice, in compute provisioned ad hoc and in engineering time spent reconstructing what produced a given model.
Data Handling Gaps
Managing very large datasets while deploying models into dynamic, real-world environments was an ongoing difficulty. The gap that mattered was the one the client described themselves: the distance between "the model works in a notebook" and "the model works on a truck" kept widening with every new utility and asset type.
Monitoring Blind Spots
Once models were deployed, visibility dropped. No robust system tracked production performance or identified when a model needed retraining - so a model whose accuracy had drifted looked identical to one that was fine, until somebody checked. That is the failure mode that compounds quietly.
Scaling Friction
Ad-hoc training runs and disconnected workflows were survivable with a small team and a narrow scope. As the client grew, every new use case multiplied the variables nobody was recording, and what had been inconvenience became a ceiling on how fast the business could take on new work.
Custom MLOps Strategy for Production-Grade Operations
The work was infrastructure around the existing models, not a rebuild of them - the same engagement model our machine learning consulting practice runs on.
Computer Vision Training on AWS SageMaker: Moved model training off Jupyter notebooks onto dedicated SageMaker infrastructure. The direct effect was cost; the structural effect was systematization - training became a defined operation with declared inputs and a recorded output.
Run Records in Amazon Redshift: Every training run logs its full context to Redshift - model settings, code versions, dataset information, resulting metrics and model weights. Nothing gets lost, and nothing has to be reconstructed from memory afterwards.
Weights & Biases Integration: Implemented and customized W&B for experiment tracking, so the team can visualize training progress and key metrics without needing direct access to any training machine. That removed the most common practical reason engineers skip running proper experiments.
Model Weights & Traceability: Weights are logged alongside the rest of each run's metadata, so a model in production can be resolved back to the training job that produced it. Without that link, deployed artifacts and training history drift apart.
Data Practice & Synthetic Defects: Shared hard-won knowledge on data integrity, error analysis, and synthesizing defective samples from non-defective ones. In inspection work defects are rare by definition - a healthy process produces few failures to learn from - so this widens the training signal without waiting for real failures to accumulate.
Production Cost Optimization: Optimized how models are invoked in production. Training infrastructure is the visible cost line; production invocation is the recurring one. Reducing that overhead changed the economics of the service, not only the cost of developing it.
Embedded Collaboration: Started with leadership - PM, CEO and data scientist - then integrated directly with the ML team, aligning on approaches task by task rather than handing over a platform. Every decision was made against a model the team was actively working on.
Deep Partnership
Invested in ground-up transformation of ML infrastructure - and counting
The Pipeline, End to End: SageMaker, Redshift and Weights & Biases
Five parts, one job each. This is a simplified view of the flow the work produced, not a prescribed reference architecture - but every stage maps to something the system now records.
1. Training Data & Job
The dataset used for a run is identified and recorded as part of it, and the SageMaker job declares its inputs explicitly: model settings, code version and dataset. Nothing is implicit in a notebook's execution state.
2. The Run Record
As the run completes, its settings, code versions, dataset information, metrics and resulting weights are written to Redshift. The record is the artifact that makes the next three stages possible.
3. Model Weights
Weights are stored alongside that record, so a model in production resolves back to the training job that produced it - which is what turns a regression from an investigation into a query.
4. Production Invocation
Inference runs through an optimized path. Training is where the visible spend sits, but invocation is where it recurs - and it is the half most MLOps projects leave untouched.
Concrete Results Across ML Operations
Significant Cost Reduction
Moving training to purpose-built infrastructure lowered compute costs, and optimizing production invocation reduced ongoing operational expenses beyond the training improvement. The saving was material; we are not publishing a figure the client has not released.
Process Systematization
ML workflows became repeatable and auditable. From training practice to model tracking and management, the pipeline gained structure it can no longer lose - which is what makes the next model cheaper than the last one.
Knowledge Transfer
This wasn't a handoff - it was a partnership. The client's team absorbed new methodologies and expertise, building internal capability that persists beyond our engagement rather than depending on it.
Production-Grade Pipeline
From notebook-based experimentation to a systematized MLOps practice, ready to absorb new utilities, regions and asset types without starting the infrastructure conversation again each time.
Computer Vision Training Infrastructure on AWS SageMaker
The move from ad-hoc Jupyter notebooks to AWS SageMaker fundamentally changed how models are developed. Training runs that were once somewhat random experiments became structured, cost-efficient operations with declared inputs and a recorded output.
Complete Reproducibility
Every training run's full context - model settings, code versions, dataset information, metrics and weights - is now logged to Redshift. This eliminates the "it worked on my machine" problem and creates a complete audit trail for any model in production. It also makes the baseline available for the next stage.
Real-Time Experiment Visibility
With the Weights & Biases integration, team members can visualize training progress and key metrics without needing direct access to training machines. That visibility shortened iteration cycles and moved model review out of one person's machine and into the team's workflow.
Sustainable Internal Capability
Beyond tooling and infrastructure, the engagement transferred deep ML operations expertise to the client's team. They now have the knowledge, processes and confidence to continue evolving their ML practice independently - which is the only outcome that survives the end of an engagement.
Who This Applies To - And Who It Does Not
It fits when the models already matter
Your models work and are already used commercially, but retraining is unpredictable, results can't be reproduced, or nobody can say how a deployed model is really performing. Typically a funded startup or an established operator with a small ML team.
It does not fit pre-product
If you are still searching for a model that works, infrastructure is premature. The notebook is the right tool until the signal is proven - building MLOps before that point buys discipline you have nothing to apply it to yet.
It fails without internal ownership
If the goal is a platform handover with no involvement from your engineers, the practice tends not to outlive the engagement. The transfer of knowledge is not a phase of the work - it is the mechanism that makes the work survive.
Governance is a separate workstream
Model cards, approval gates and audit exports for regulated automated decisions build on reproducibility, but they are policy work rather than infrastructure. Worth scoping separately, and worth doing after the record exists.
What Makes MLOps Work - And Where It Usually Stalls
MLOps is not a tool purchase. It is the order in which reproducibility, visibility and automation get solved - and most teams get that order wrong. It also has a running cost: engineering time moves forward, telemetry accumulates, and discipline has to be enforced rather than assumed.
Reproducibility Comes First
If you cannot reproduce a model, you cannot improve it deliberately. Every retrain becomes a guess about what changed, and every regression becomes an investigation instead of a query. Everything else in MLOps is built on this one property.
Monitoring Needs a Baseline
A drift signal means nothing without a record of what the model was trained on and how it measured at the time. The baseline is the monitoring - without it, alerting is guesswork with extra steps, and retraining decisions get made on intuition.
Automation Comes Third
CI/CD from commit to production belongs after reproducibility and monitoring. Automating a process that is not yet reproducible or observable just produces unreproducible models faster - and makes the eventual cleanup more expensive.
The Team Has to Own It
MLOps that nobody inside understands tends not to outlive the engagement. A tracking system is only useful if every run goes through it - one engineer training locally "just this once" reintroduces the exact problem the infrastructure was built to remove.