Observability

Upon joining, I came up to speed quickly on the stack of the early version of Pull Workbench, which was very buggy but demonstrated the initial ideas and had a solid set of the latest technologies and patterns established in the codebase, providing for a solid starting point.

I was entrusted to aid our CTO in hiring several additional employees, and so I joined and conducted interviews for the first several months while working with the existing team AI + Full Stack to deliver features and solidify the system, with the aim of keeping it fully working with each merge, after playing a little catch-up to fix the early bugs that worried our business partners, giving them confidence that our team could deliver.

From there, I developed full stack features solo or by pairing with team members, and ultimately led a squad of 5 team members alongside a second squad that together comprised our engineering team.

Much of my time went into authoring complex analytics sql queries using the impressive Kysely library, a fluent, typesafe query builder that we used for our postgres and redshift databases. Given the nature of the product, we needed to make decisions on which queries could be run in real time vs. which queries and subqueries would need to be computed offline as part of a network of airflow dags.

On the ML Ops side I advocated for traceability and reproducibility / determinism of all models and artifacts, and integrated with systems that implemented that, such as Airflow to coordinate DAGs of ML training jobs and Sagemaker's metadata API, which we controlled via model lifecycle automations that produced and stored models, artifacts and metadata that were in turn consumed at runtime or in batch by our analytics stack

On the frontend, I helped us deliver an initial version of the Pattern Editor, a UI and set of APIs that users could use to put together their own patterns of interest, such as looking for certain anomalous ranges of quantities that themselves may be derived from other user-defined patterns. This entailed not only a UI that was DAG-aware but also a layer that converted the json representation of these patterns from the frontend into typesafe kyesely queries to be executed against redshift.

The candidate summary page summarized a candidate's performance during multiple interview stages by presenting radar charts showing degree of fit against the values and attributes being evaluated for their position, as defined in the Interview Builder.

I built the frontend in React and Typescript, and integrated with the backend, which I partially built, which leveraged RAG and ran several Machine Learning models to produce scores and explainable AI. For example, models to break down interview transcripts into quotable fragments, evaluate relevance against configured company values, and call chatGPT APIs to obtain summaries and scores related to that content

I ran Appen's ML Platform, which was used by FAANG and many other startups and enterprises to automate and scale their ML practices, including running both supervised and unsupervised workloads, as well as their global annotation workforce which enabled customers to leverage our crowdsourced professionals to elastically obtain labelling and quality checking services for text, voice, image, video and LIDAR annotation, training and validation use cases.

I reported to the CTO and directed multiple full stack teams each with their own tech leads and range of engineering skills to do both regular maintenance and product enhancements using technologies like Sagemaker, React, K8s (Kubernetes), Spark, Kafka, Airflow, Spring(Boot), Ruby, Python, Java, Typescript and SQL.

Maintenance included regular updates to infrastructure, bug fixes, and performance optimizations across the platform. We migrated more and more services to K8s (Kubernetes) and Ambassador as our API gateway, where we could consolidate cross-cutting logic like auth and versioning.

Enhancements included changes to simplify the UX, kill redundant or unused features, add measurement to inform our choices, and larger efforts like Enterprise OAuth.

There were 4 different websites in different technologies, acquired from different companies, and some APIs, that all needed to be unified in terms of sign up, sign in, and sign out, given their existing state of each having separate user stores, including 3rd party vendor users who logged in with vendors and then authed to us with a hidden token.

It was a stalled project, so I started with missing requirements, incomplete designs and misleading progress indicators and focused other leaders and teams on delivery thru tested working software, focusing on tested user stories and on-the-ground learnings as units of progress, instead of large, outdated PRDs waterfall style.

Contributed directly in React / Typescript, Nodejs / express, Ruby on Rails and custom gems, OAuth configuration, Java Spring with runtime loaded SPI implementations from across separate applications domains.

There was a complex architecture at play and teams that did not know each other and weren't working as a single unit, so the landscape was difficult and rife with demoralized team members.

Although my team was to play but one part in many on the project, I realized quickly that there was no single leader or coherent plan, and so there was lots of blame game and treading water.

With permission from our VP of Engineering, I took charge of the teams and worked with product to firm up requirements, and replace the initially conceived solution architecture, which would not have worked and was created in a bit of a vacuum, into one that would actually work, by digging in and running all the services and web apps myself and understanding the multiple data stores and existing auth mechanisms including auth via 3rd party vendors to some parts of the system.

I delivered the project within 5 months and for my efforts was rewarded not long after with a promotion.

My PM and the business wanted to illustrate to other teams that a fast-paced, fail-fast approach where we released daily (as opposed to 1-3 times per year) would serve us much better in that we could learn quickly, iterate and pivot, without huge costly investments into products that did not meet expectations or deadlines.

Before hiring my team, I set up a CICD pipeline and basic framework of a site that could sustain a heavy and intense crawl from google.

New hires all released to production on their first day of work - a principle I had brought to the table, that it should be so automated and simple that someone could set up and deploy a small feature within their first few hours of working at Sourceability.

Our parts and datasheets website, which also incorporated proprietary availability and quality scores, was used - within 3 months of inception - to successfully sell a 3 year Analytics API contract to an international multibillion dollar company, as well as driving organic traffic and learning how to scale to sustain google crawls of the hundreds of thousands of electronic component parts in our inventory while scaling down outside of the crawl / high-traffic moments.

• Full Stack - React, NodeJS, Typescript, Kubernetes, Gitlab
• Functional Reactive Programming - RxJS, highlandjs
• Daily Production Deploys - Canary Deployment w/ K8s
• Constant Collaboration - No “throwing over the wall”
• CI/CD Automation Pipeline - Every user story gets an instant shareable environment
• Coaching / Mentoring / Leading diverse team

A portal bringing together version control, automated test definitions and statuses, quality metrics, jira tickets, CICD jobs, and supportinginfrastructure definitions and status into a single place to aid in release management and devops practices.

Behind the scenes, pipelines made with K8s (Kubernetes), Mesos, Github and Jenkins automatically provisioned environments, deployed our software and ran extensive tests on it, including complex multi-cloud platform scale tests across Google Cloud and AWS as well as on prem with bare metal and Open Stack

I initiated and led "Scala Dojo" ,partnering with QA / Devs interested in adoption of Functional Programming, leading to certifications from Coursera and improved team morale and interest.

That broadened and continued on as "DevOps Dojo", which was a recurring collaboration initiative that produced theDevOps Portal + CICD by engaging across teams and disciplines to ascertain true priority pain points and solutions that scaled across multiple teams, so that we could address those via CI/CD and our DevOps practices.

Observability was introduced to MapR via the Spyglass Project, which sought to obtain metrics from workloads as well as application specific metrics across all the tools and infrastructure of the MapR Hadoop Stack.

My responsibilities included automating the build and deployment of the full hadoop stack under development, automated test authoring and execution, mentoringjunior teammates to do the same, collaborating with dev teams to ensure they plugged into our CI/CD and Test framework nicely, andtroubleshooting problems that arose.

• Introduced Scrum process and Jira to the team
• Acted as scrum master in early days of project
• Developed automation to manage ESXI environments for CI/CD, involving heterogeneous MapR cluster installation, configuration and testing

New Admin Dashboard - Architected and implemented green field admin dashboard, worked with junior devs and leadership to bring about in the requested stack: Bootstrap / JQuery / Ruby / Java / MySQL / LDAP. Users used Admin Dashboard to administer Organization, Role, and other management aspects of the Explorys EPM Suite.

Dependency Injection Framework - To enable high testability, all components required dependencies to be passed in, and so eliminating factory boilerplate without adopting a complex DI system was desirable. Wrote and shared a simple runtime-reflection based DI framework with multiple projects / teams

The Quoting (F3) was a success, but to roll out to all 50 states, we needed an engine that could handle the complexity of the system, render quickly, and that was easier for our engineers to build and test with.

I took the most complex page of the Direct Auto Quoting app - the "Buy Page" - and drastically improved performance, reducing render time from 28 seconds to 2, by isolating the page and building it with a prototype of what would become REF2.

Seeing such a drastic improvement in performance gave the business the confidence they needed to convert Quoting (F3) to REF2 and use it to roll out to the remaining states. (sidenote: a code-oriented framework, FlashQuoting, which did away with REF2's markup and code bindings, superceded REF2 before all 50 states were rolled out. The later popularity of and similarities with React confirmed REF2 was onto something.)

Technical Details REF2 targeted both Flash and HTML during the pre-webkit era. Created technology-independent language and APIs for describing UI hierarchies, cascading styles, business logic and arbitrary data structures in a way that abstracted the developer away from the details of the client/server event communications, marshalling between multiple client technologies, persistence of state concerns, or the details of rendering engines / APIs in the various supported environments. Aided in development of dev tools such as code hinting and code generation to facilitate quick onboarding, compile time checks, type safety and debugging.

Having shown in Quoting (F3) and REF 2.0 (UI Framework) that tests and low complexity policies led to fewer bugs and higher velocity by doing some a/b studies, the business invested in tooling to automatically measure and enforce policies across the Direct Quoting line of business software development teams.

These teams used a variety of software stacks and technologies, including proprietary build and release systems.

I consulted with the build and release team, at the time, a separate team, to understand their plugin architecture which already was in use for all said teams.

Next, I worked with each team to understand their level of code coverage across different types of tests (unit, integration, end to end), and agree with them what targets they wanted to meet, and at what point in time.

Some teams required me to implement coverage measurement tools. Keep in mind that we were not able to use open source tech, and actionscript had no unit testing or code coverage tools.

I worked with a mentor of mine to write a lexer-parser-generator, which took the grammar for the actionscript language and allowed us to instrument our codebase with a pre-build step that inserted beacon calls with metadata into the various methods of the application. The coverage monitor and unit test UI was written in C#.

I exposed the coverage reporting service using web services running on SOAP via MS ASP.net WCF communications stack. Initially an MS SQL database housed the data, but as it grew, we moved it to a data cube where various dimensional summary and BI queries could be more efficiently run without interfering with the transactional nature of concurrent builds reporting their metrics in across the business.

Built upon existing code analysis and instrumentation tools to create a cross-platform solution for the build-time analysis of unit and system test code coverage, cyclomatic complexity, code coupling, defect density, change volume, maintainability, and other quality metrics. Data from many different proprietary formats is transformed into a single canonical format, where it is in turn normalized into a relational structure to facilitate on demand querying for system-level quality benchmarking, real-time code quality reporting, providing objective insight into QA risk assessments / test strategies, and enforcement of architectural standards and constraints.

Progressive had two sites for Direct Auto Quoting - one used by customers at home and the other used by our call-center reps. Given the complex state-by-state variance in the insurance laws, this made for a huge maintenance cost, and doing it twice in two codebases didn't make sense.

The premise was that a single web 2.0 Direct Quoting Application could replace these while also yielding a much more modern and customer-delighting application.

As the team's Actionscript expert, I joined and quickly helped out delivering feature after feature, and I loved the XP discipline and grew to appreciate TDD especially after the site started hitting performance problems that warranted significant refactors, which would have been much riskier without test coverage!

Our pilot included 14 of the 50 states, and was a complete success. However, the amount of clientside rules and assets started slowing the app down, and this was when I was asked to replace the REF framework that preceded my joining, with something much faster and more developer friendly. This led to my promotion to lead engineer, where I begun work on REF 2.0 (UI Framework)

montgranite.com - supplier of natural stone website. After assessing their products, we categorized them according to stone type, colors, texture and brand, then I created a MySQL database and an ORM to read/write PHP objects and their relationships from the database and present it as a front-end, which I designed to resemble a piece of marble.

I then hosted the company's website and email for a number of years afterward and actively updated the site on retainer.