Production stories, told the way I would tell them across the table at an incident review: the scale, the failure, the diagnosis, the fix. Employers and clients are left out on purpose; the engineering is not.
KubernetesRancherHPANginxRedisGrafana
Clients → gateway → pods, each with its cache
The scale profile
Aggregate volume above one billion network requests a day across more than thirty dependent services.
A single public-facing gateway sustaining 450 million requests a day to outside clients.
Autoscaling pods under Horizontal Pod Autoscaler thresholds I tuned against real traffic, not defaults.
Objective: sub-millisecond execution loops, zero-downtime hotfixes, and fault isolation between services.
The architecture
State is kept out of the runtime containers. Ingress lands on a hardened reverse-proxy layer that routes by path into isolated pods, each service with its own cache pool and its own scaling curve. I built the gateway before the current wave of AI tooling, became the person the company relied on for that part of the system, and kept refining it as the tools improved.
What AI changed
The intuition came first. Once language models matured, I wired them into the diagnostic loop: scripts that feed scrubbed log blocks and metric queries to a model and draft the hotfix, while the judgment stays with me. Years of knowing where a system breaks became something I could hand an agent: a force multiplier, not a crutch. The same idea later became a product of mine that lets an AI read a cluster's logs and metrics directly.
What it taught meAn autoscaler's defaults are somebody else's traffic: the thresholds belong to the architecture, tuned against the requests you actually get.
PromQLGrafanaDynatraceKubernetesForensics
Working set against the limit, then the kills
The incident
A mission-critical gateway began dying in abrupt termination cycles at peak traffic. Application logs showed nothing: no leak, no unhandled rejection, no error before the container was killed.
The investigation
The platform team attributed it to application memory. I bypassed the dashboard summaries and wrote raw PromQL against the cluster, charting each pod's working set against its limit alongside the heap the runtime itself reported.
sum(container_memory_working_set_bytes{container="gateway"}) by (pod)
/
kube_pod_container_resource_limits{resource="memory", container="gateway"}
The root cause
The limit was being hit by memory the application never allocated. The platform tier had injected a third-party monitoring agent into every pod out of band, and a minor release of that agent carried a regression that blocked under high concurrency, amplifying memory until the orchestrator stepped in and killed the pod. Nobody, including the vendor's own support team, had connected the two.
The fix
A container configuration override in our pipeline that pinned the agent off the faulty version, deployed the same day.
Autoscaler metrics restored to organic, predictable steps relative to real demand.
The metric traces and dumps handed to the vendor's platform team, which accelerated the official patch.
Full availability restored with no change to application code.
What it taught meWhen the logs are silent, chart the raw working set against the limit: the memory that kills a pod is not always the application's.
PaymentsPCIFraudMicroservicesAWS
Processor → ledger → reconciliation
What was built
Payment-processor microservices that accepted more than ten million cryptocurrency payments, integrated with card processing, adding over two hundred thousand dollars a day in revenue at peak.
The company's Tier 1 PCI compliance, led end to end at the six-million-transaction level.
Dashboards for the fraud and accounting teams that materially reduced card chargebacks.
A social chat product on the same platform, and new card processors added to the live payment system without an outage.
Why the accounting degree matters here
A ledger has to balance, and every mutation has to be traceable. I write payment code with that frame: idempotent operations, reconciliation that runs unprompted, and no hidden state anywhere a dollar passes through.
What it taught meTreat payment code as a ledger: every mutation idempotent and traceable, and the reconciliation running before anyone asks for it.
A service in a storage and delivery platform started crash-looping in production. Every dashboard said the pods were fine: no heavy CPU, no climbing memory, no slow responses, right up until they went down.
Holding the line
The first job was keeping people served. While I worked on a real fix, I kept traffic moving by increasing capacity and shifting it between our two data centers whenever one side started to fall over. That bought the time to do this properly instead of quickly.
The investigation
When every resource graph says healthy and the pods still die, I stop asking how much they are using and start asking whether they are free to work. A Node.js service handles its requests on a single event loop. If that loop blocks, everything queues behind it even while CPU and memory look completely calm.
I immediately pulled in prom-client to measure the runtime telemetry directly. The data proved that our application layer was handling the execution cycles perfectly. The bottleneck wasn't our services; it was a resource-intensive regression introduced by an environmental infrastructure change.
An outside platform team had injected a minor release of a third-party monitoring agent into our base images, running as its own binary on the underlying Linux system. Under peak concurrency, this companion process began choking on string processing and data serialization back to its external servers. By auditing our raw traffic, I discovered that the agent was natively capturing incoming request headers, including standard, massive JSON payloads sent by certain clients that our services safely ignored. Because the agent operated out-of-band and processed these entries raw, it created a massive processing bottleneck on the host that starved the container and crashed the pods.
The fix
I built log trimming in front of the logging system and extended it to sanitize incoming request headers. Stripping out large unused client headers before they reached the monitoring tier dropped a representative JSON log entry from 81 lines to 31.
Isolated the environmental failure by proving that shielding the injected agent from high-overhead string processing completely stabilized the pods.
Proactively surfaced additional metrics from prom-client to Kubernetes via the custom metrics API, expanding the HPA to support scaling on runtime metrics as a long-term architecture safeguard.
Proven end to end in test with our QA engineers before anything moved: smaller logs, and pods scaling on both lag and CPU.
Getting it to production
Production ran an older release than stage, so the fix had to be cut from the production version, and our pipeline only built and published an image after a merge to main. This was our first production issue of this kind, so I worked out how to build and version an image from a hotfix branch, and I told the teams that rely on us what it meant before it went to stage: for a while, stage would be production plus the fix, without the newer features. Then I wrote the production change request into the change management system.
During the production deploy the new pods would not schedule. Twenty-one of the twenty-four nodes had no CPU room left, and the other three were control-plane nodes the pods could not use. I scaled the pre-deploy pods from 80 down to 44 to make space, the rollout finished, and the change closed clean.
Closing the loop
An emergency branch that lives forever is the next incident waiting. I opened the follow-up work and brought the log sanitizing, the metrics and the rest of the hotfix forward onto the main line, so the next release carried the fix instead of quietly dropping it.
What it taught meA healthy runtime can still be taken down by a greedy sidecar: true system visibility requires separating application execution metrics from host-level processing bottlenecks.
More than 280,000 work items through the portal so far, with the share handled automatically growing from about one in five to about two in three.
In one year, over 5,000 new hires onboarded and 560 other requests handled, each onboarding granting anywhere from 5 to 75 separate pieces of access.
Rosters of up to 99 people submitted as one batch.
How it helped the organization
Before the portal, a manager filed a separate request for every piece of access for every person, and each one waited on its own email approvals from a manager and a data steward. The portal replaced that with personas: role templates their owners approve once and then delegate. A new hire's access comes from the persona, already approved, and every grant is recorded.
That record is what auditors use. When someone transfers, they look up the portal's tickets to confirm the old access was removed before the new access was given. Before, the same answer took hours of digging through a text-based system. I have carried this portal on my own, and it is some of the most quietly useful software I have worked on.
Moving it to a new ticketing platform
The ticketing system underneath announced its retirement, and the replacement thinks differently. The old API treated a ticket as one flat record; the new one is a tree: a request, the items in it, and the tasks under each item. Swapping endpoints would have broken every workflow that reads a ticket.
So I mapped how the three levels join, worked through which data a manager should actually see, and built the client around the new shape. Ordering there is asynchronous, so the portal keeps the request number it gets back and checks for the items and tasks, rather than placing a second order when the first one is slow.
Keeping it secure and current
I introduced fully automated container rebuilds across a fleet of 30 services under my control, ensuring security patches no longer waited on manual intervention. This initiative dropped open container vulnerabilities from dozens to single digits within the onboarding platform, and from several thousand to under 200 across 24 companion services.
Migrated 15 of these services from legacy SQL logins to secure Windows (Kerberos) authentication, while identifying, consolidating and retiring older obsolete services to shrink the attack surface.
Critical services moved to TypeScript, and the stack brought up to current React and Bun.
Twice-yearly access attestations and the reports auditors read, transfer dashboards that say what each ticket is for, and admin controls for queues and timers that change without a release.
Catching it before it is an incident
When worker nodes were being added to the cluster, I found the shared NFS volumes that would be affected across six projects and worked it through with the storage team before anything broke.
When the domain controllers were replaced, I showed the services read their Kerberos settings from a cluster ConfigMap, so the fix was a config change by the right team, not a code release.
A deploy pipeline that kept failing on a hard-coded path in an older script: traced, and fixed so the path comes from the environment.
Two high-severity security findings shown to be false positives: the flagged query parameters never carried an insecure payload.
What it taught meAccess is a ledger too: when you grant permissions from approved templates and record every step automatically, a compliance audit shifts from a frantic forensic investigation to a simple verification task.
KubernetesRancherRKE2HarnessMigration
One pipeline, two clusters, a way back
The job
The services ran on Rancher's RKE1 and the platform was moving to RKE2. The fast version of that is one big cutover night. I wanted a version where nobody had to bet the estate on a single deploy.
The approach
Harness deploys every service to both platforms through separate pipelines, so RKE2 is proven while RKE1 keeps serving.
Development first, and only development, until it was fully tested. Test, stage and production each follow as their own step.
Moving traffic is its own step, separate from deploying, with routing back to RKE1 written into the plan in case validation turns something up.
Production is gated: no work starts there until every non-production environment has moved, and it too runs on both platforms before anything relies on RKE2.
The unglamorous part
Most of the work lived outside the application repository. I designed token strategies and wrote orchestrations against the Rancher and Harness APIs to automate cluster registration, inject custom configuration overrides, provision RBAC permissions, and sync core secrets across environments.
Beyond the automation, it meant doing the groundwork that makes a blueprint real: tracking down cross-team dependencies until the new network connectors actually existed, and carefully restructuring the ingresses and networking layers to enable zero-downtime traffic swapping.
The outcome
The migration concluded with zero downtime. The entire application estate is operating fully on RKE2 today, having moved through test, stage, and production in isolated, sequential phases that kept a verified rollback path open at every step.
What it taught meA migration you can reverse is one you can actually start: by running both platforms side by side and moving one environment at a time, we kept the old road open until the new one proved it could carry production traffic seamlessly.
I engineered this cloud platform to run commercial B2B SaaS products with near-zero operational overhead, decoupling high-performance edge computing from cost-effective on-premise storage. The goal was enterprise-grade data privacy and absolute uptime without a dedicated operations team.
Every app is one Cloudflare Worker with its own independent database and a stage tier that holds no real accounts. Sign-in is managed by a single identity provider behind a zero-trust gate. Media lives in an edge-cached byte store that falls through a secure tunnel to local hardware, resting large tenants in resilient object storage. Nightly jobs pull every database home, encrypt a snapshot of the state root, and push it off-site.
The runner
Work enters as a Markdown file in a queue. A launchd job wakes an agent, which claims a task, makes the change, runs the repo's gate, deploys through one locked script, proves the live commit and files whatever it could not finish as a new task. A failure mode, once found, becomes a check with a test of its failure path, never a warning in a document. The site you are reading was built and deployed this way, and so are my three products.
What it proves
Commercial multi-tenancy can be perfectly isolated at the database query level through strict software gates, removing the need for expensive infrastructure fragmentation.
Disaster recovery doesn't require complex orchestration; minimizing system state into two directory roots on disk makes backups fast, ironclad, and easily verifiable.
Infrastructure-as-Code registry structures can eliminate environment drift entirely, keeping every host, job, and cloud resource accounted for automatically.
A solo engineer can scale and defend a production estate normally requiring an entire DevOps department by offloading execution to autonomous agents.
What it taught meAn agent works unattended safely only when every rule it must follow is a check it cannot pass by accident.
Designed distributed head-unit registration flows and secure OAuth authorization gateways for third-party service access.
What it taught meA device in the field is a detached system state: remote actions must be engineered as entirely asynchronous, non-blocking workflows from day one, never retrofitted as an afterthought when a synchronous network thread inevitably hangs.
RasaNode.jsReactRedisMongoDBKubernetesAWSMCP
Portal → session, intent, reply → engine
The evolution
I architected conversational platforms across two distinct eras of AI capability. The initial platform utilized an open-source intent engine to replace dense UI input forms with structured conversational pipelines. Later, this evolved into an enterprise-wide language framework servicing extensive organization pools with dynamic file execution and multi-model routing capabilities.
Built intent-parsing engine abstractions comprising custom token handshakes, session pinning in Redis, and database-driven PDF document rendering pipelines.
Designed stateful runtime session managers and integrated Model Context Protocol (MCP) servers to chain external analytical payloads natively into agentic workflows.
Rebuilt the primary runtime interface utilizing modern React to facilitate parallel multi-variant testing across distinct model backends under volatile load spikes.
The infrastructure incident
Production pods were restarting several times a day at hundreds of millions of requests across the region, and the first guess, as it usually is, was a software memory leak.
The baseline metrics proved otherwise. Heap allocations sat securely at 15 percent, but host CPU bounds were continuously pinned at 100 percent. Under sustained heavy computational strain, the runtime process stopped acknowledging container liveness probes, triggering an automated Kubernetes SIGKILL termination sequence. The pods weren't leaky; they were structurally starved for compute.
To stabilize execution, I isolated the high-concurrency bottleneck, coordinated right-sized core processing boundaries with the platform teams, and established explicit TTL index migrations to cleanly cycle long-form database context trees before history bloomed into overhead.
What it taught meThe model decides what a message means; the service around it decides whose conversation it belongs to, and that is most of the engineering.
Business portals let customers upload their own spreadsheets and CSV files, and the analytics behind those portals queried Kusto. I built the ingestion system between the two, as Azure function apps.
The shape
An upload is taken by a function app that reads CSV and Excel into rows the warehouse accepts.
The rows travel as messages over event hubs and a service bus, so accepting a file and loading it never wait on each other.
The warehouse loads them, and the portals' analytics read them from Kusto.
The hard part
Customers' files ran to multiple gigabytes. A function cannot read a file that size into memory and then decide what to do with it, so it streams: rows are parsed as they arrive and sent on as messages, and the memory in use stays flat however large the file is.
It was not my first pipeline of this kind. An earlier one carried flat files into Kusto and Databricks behind REST APIs that checked every request against a schema before it reached anything internal.
The team around it
I set the testing baseline for these services, with Mocha for the tests and Nock standing in for every external call, so a test never depends on someone else's system being up. I also did a lot of the code review and spent real time with engineers who had just joined, walking them through how the system scaled. That part of the job I would happily do more of.
What it taught mePut a bus between accepting a file and loading it: the customer's upload is finished when the file is taken, not when the warehouse is ready.