The full story

Most of the difficulty in operational data shows up before the model does. Signals are noisy. Labels are scarce, or they come secondhand from a process nobody designed with modeling in mind. And the data you fit last week isn’t the data running today. I spend most of my time on that end of the problem, and then on building the systems that carry whatever survives it into production and keep it working once it’s there.

Right now I’m a principal engineer on a greenfield platform for anomaly detection, correlation, and forecasting, built for roughly 32 million broadband endpoints. I own the modeling and the design of the cloud architecture underneath it, and I work with leadership on where the platform goes next. The current answer is agents that act on what the platform finds, so a lot of my recent thinking has gone into what those agents should and shouldn’t be allowed to do.

The hard parts there have surprisingly little to do with picking a model. At 32 million devices, a separate model per device isn’t practical, and a single global model only notices the biggest failures. So I score each device against comparable devices in comparable operating states. Daily load cycles look like faults to a fixed threshold, so seasonality has to live inside the baseline. And a five-minute cadence across a fleet that size puts a hard ceiling on how much computation any one device gets, which turns feature selection from a preference into a budget. Labels were the other gap. Every anomaly the platform flags now goes to an analyst who confirms it and adds context, so the ground truth gets stronger the longer the system runs.

I rarely use a modeling framework as packaged. I’ll read through the source of something like Prophet or AutoGluon to see how it handles one specific piece, then write my own version of just that piece for our data. An automated tuning pipeline then keeps the parameters current as the data shifts. Choosing the method is most of the work, and it depends on knowing where each one stops holding more than on which one is newest. Regression and tree ensembles are often the right answer. Deep sequence models earn their place when the structure calls for them. The fitting, regularization, and validation around that choice gets less attention, but it’s usually what separates a model that looks good on a holdout set from one that still works six months later.

Before this I spent close to a decade in consulting and applied data science across financial services, insurance risk, and industrial commodities. Every project meant learning an industry well enough to understand what its data was saying, then building something that fit it. The most useful thing I took from those years: how to take a question nobody has phrased clearly yet and make it measurable.

My side projects follow the same thread. Tracking targets without GPS by fusing RF, thermal, and telemetry data (SENTINEL). Catching failing satellites before they fail (CONSTELLATION). Coordinating drone swarms with multi-agent reinforcement learning (HYPERION). Different domains, same question underneath: what can a corrupted signal actually tell you?

I came into this through finance and risk and went on to a master’s in applied mathematics. Most of my reading since has gone into foundations, lately optimization theory and information theory. That’s less abstract than it sounds. Information theory tells you whether a signal can support the detection you’re asking of it, and nothing downstream recovers what an upstream step already threw away. Optimization theory tells you whether the problem you’ve written down is well conditioned before you spend a quarter trying to solve it.