#13 - How we set a technical direction that lasts (and stuck to it!) | Andrew McVeigh, Chief Architect at Suvoda
Aligning the (North) stars
Hello there, welcome to the 13th edition of our series of in-depth articles, where we invite world-class engineering leaders to share their wisdom.
This week, we’ve invited Andrew McVeigh, Chief Architect at Suvoda, and former Chief Architect at LiveRamp and Hulu, to share his tips on how to set a technical direction for your organization and the common pitfalls to avoid.
But first…
Advertising break (…kinda)
This post is sponsored by Plato Elevate (aka, ourselves 😅), the No-BS conference for engineering leaders. Join us in San Francisco on June 5th-6th to get actionable, no-fluff insights from world-class engineering leaders like the creator of Google Street View, CircleCI’s CTO, Coda’s Head of Engineering, or Stripe’s Head of Payments and Engineering.
P.S.: We’ve extended our early-bird pricing — get 58.3333333% off your ticket now (or never).
Andrew McVeigh, a 30-year career in tech
Andrew McVeigh is a seasoned Chief Architect, with a knack for solving complex problems across industries spanning from gaming to digital broadcasting or clinical trials. He’s played key roles at companies like Amazon – where he worked on Prime Video – and Hulu. For the latter, he and his team were the ones who made sure you could watch the Super Bowl and March Madness without interruption. Before that, he worked for Riot Games, the editor of League of Legends, the world’s most-played game, and designed the architecture of the largest project undertaken in 5 years there: the League game client.
Andrew holds a PhD in software engineering from Imperial College London, where he dove into how to create fully extensible systems. You’ve guessed it, his whole career revolves around building technology that lasts – and grows. Now a Chief Architect at Suvoda, he revolutionizes clinical trial software, making it more adaptable, patient-centric, and scalable.
We’ve had the opportunity to see Andrew on stage at Elevate’23, where he shared his master plan to set a technical direction that lasts in any organization, avoiding the common pitfalls and navigating management challenges. The below article is a deeper-dive (yep, I just coined that term!) into his talk, where he goes into more detail on his framework. We’re sure you’ll get a ton of value from it!
You can follow Andrew McVeigh on LinkedIn. Let’s go!
How we set a technical direction that lasts (and stuck to it!)
Setting and adhering to a technical direction is critical to the success of an engineering organization—and I should know. I’ve seen a number of cases when it wasn’t working and have also been fortunate enough to increasingly experience when it goes right.
I’m Andrew McVeigh, Chief Architect at Suvoda, and I’d like to share my key observations and learnings about how to set a technical direction.
First off, a quick introduction: I’ve been working as a lead or Chief Architect for the past decade at companies like Riot Games, Hulu, LiveRamp, and Amazon before joining Suvoda about three years ago.
Suvoda is a ten-year-old company that produces a SaaS platform that supports complex clinical trials. There are three main aspects of our growing ecosystem:
eConsent, an interactive and engaging informed consent experience.
Interactive Response Technology (IRT), an advanced clinical trial system that manages patient visit windows, drug randomization, and many other logistics. The typical trial is double-blinded & randomized where you don’t know if you’re getting the placebo or not and neither does the physician administering it.
Electronic clinical outcome assessment (eCOA), which assesses whether or not the treatment is working via a set of clinical questionnaires.
Our software can support complex and varied study requirements, offering flexibility to configure or customize the software as needed to meet the needs of the applicable studies. This means that we never have to refuse a clinical trial because our current software can’t handle it. We’re equipped to handle any request a complex trial throws our way.
Suvoda’s Architecture team is currently quite small—as it should be. My strong belief is that architects should be in the code and not just drawing boxes and lines.
I see the interplay between product, architecture, and engineering as a table with three legs—product decides what is built and in what order, architecture does critical design and gets engineering consensus for how it’s built, and engineering executes on it while also doing architecture and design. By growing the engineers’ architectural skills we envision one day getting to a space where the architecture team is either not needed or can be very small.
Identifying the need for a new technical direction
Every company gets to a point where the success or pivoting of the product line means you have to address the architecture. The architecture that got you this far may not be enough to get you further.
At Suvoda our goal was to set up the technical foundation for the next five to ten years, and to support the products—to create a strategic platform for a multi-product future. In the rest of this post, I’ll share some of the ways we were able to make this transition successfully as well as the pitfalls I’ve seen occur in the past at various companies.
The happy path: How we achieved success
In this first section, I’ll share some of the things that went well at Suvoda—our “happy path”—to help you consider how you might apply these learnings at your organization.
Step 1: We got clarity on product direction
I joined Suvoda to help the company through the next step of its technical journey. At the time, Suvoda was at an inflection point where they’d been successful, but now they needed a technical reimagining because of that success.
One of my first tasks was to ensure clarity on the product direction, getting answers to questions like where the company sees itself in five years, what its vision is, what the CEO and sales team care about, what the users value, and what the CPO wants to see achieved. This step was critical because it’s dangerous to make technical choices and move in a technical direction that’s not helping the product.
Through this process, we discovered the following:
We needed to retain the powerful level of customization we pride ourselves on. If we didn’t have this, we risked losing our customers requiring more complex trials.
We needed to increase the efficiency of customization.
Products needed to be flexible to enable us to add new features rapidly.
We needed to enhance our SaaS platform and ecosystem to allow us to quickly introduce new products on the platform.
Step 2: We created an ambitious North Star architecture and devised a conservative way to get there
Your North Star defines where you want to go, which is critical when guiding a team. But it’s just as important to devise a conservative strategy that charts how you’ll get there. “Conservative” can mean different things at different companies—sometimes it means moving fast, but trying not to break things along the way. Because Suvoda is a software company that deals with clinical trials, breaking things is not an option for us.
We landed on: “A powerful way to manage a set of highly customized systems as a multi-tenant SaaS platform” as our North Star. This would enable us to create a platform that can deal with the large variability efficiently, addressing the natural tension between “use our standard system” and “yes, you can have all the customizations you’ve asked for.” I did my PhD in software extensibility, and I drew heavily on some of the learnings from that.
When defining your own North Star, I recommend keeping these three points in mind:
It needs to match up with company ambition: If your North Star is not ambitious enough, your CEO and board will feel like they need to bring in someone else who sets more ambitious goals.
It needs to be achievable by the team: Consider what your current team is good at and capable of. One of the worst situations to find yourself in is to have a North Star that can’t be accomplished.
You need to be prepared to sell, communicate, and over-communicate the vision: Get comfortable sharing it widely to get others on board.
As mentioned above, we didn’t just come up with a North Star and consider our work done. We also devised an incremental approach to get there. This is important because you want to prove that you’re making valuable contributions as you progress toward your destination.
In our case, we found our “center of gravity” by focusing on what we do really well. We decided to shut down parallel development, focus on the core system, and add APIs and low-code/no-code features. This gave us a strong platform to build out additional products for our ecosystem.
So in our case, we brought the core parts of our system into a single development train, creating a powerful foundational platform. We unified our API across the entire product suite, aligned to our strong and consistent internal CQRS architecture. Finally, we added no-code/low-code primitives to enable us to work efficiently.
Let me go a bit wider than just Suvoda and discuss a key anti-pattern I’ve seen companies adopt while trying to reimagine their systems. For instance, it is very common to start up Team A on a next-generation system, while keeping the “legacy” / Team B working on the older system which is earning all the revenue. This never works well in my experience. Team A has to play catch up with the ever-evolving legacy system, and the situation comes to a head after about three years which involves shutting down Team A. In the meantime, Team B can become disillusioned and leave.
To avoid this, we need to focus instead on incrementally improving and bringing the current system into that North Star architecture. It’s usually possible to do this far more efficiently than people realize. The cost of parallel development is also far larger than people realize - it’s essentially an attractive mirage.
Below is a schematic showing how I’ve seen companies reach the desired target state without disrupting their critical systems. Evolution is often more powerful than revolution, particularly when you are starting with a system that has more strengths than weaknesses.
Step 3: We communicated transparently
The next step involved communicating the direction and value throughout the company. This involved a few tactics, including:
We published each architectural decision as a request for comments (RFC).
When I was working in investment banking IT, I saw the power of describing what problem was being solved, outlining the requirements and approach before full development was started. I’ve seen this work at a massive scale, with 20k employees or more. It makes sense because everyone can comment on whether the development is being approached in the correct way. We have previously called this document an Architecture Check at Definition, but others know it as a Request for Comment (RFC).
RFCs are documents with one owner and a standard set of headings that force the designer(s) to consider the fundamental engineering aspects. The headings are things like “summary”, “requirements”, “overall design”, “why are we building this system?”, “alternatives considered”, “risks” etc. It never hurts to be reminded of the fact that we need to know why we are building and what constraints we have before we go ahead and invest a lot in development.
I can then look across all the RFCs at once to see the state of our architecture. Each RFC includes a list of stakeholders who have to sign off on it. While you may need to wait longer with certain stakeholders, eventually once you get everyone’s sign-off, you now have a document that proves that all stakeholders had the chance to weigh in and raise their objections. Once they’ve signed off on it, they’ve signed up for it. It forces clarity.
We also collect data from every stakeholder involved in the RFC, asking them to rate their agreement with statements like “The core proposition is the right approach”, “The North Star has the right level of aspiration” or “I agree with the way this system has done X”, etc. This way, we can identify potential problem areas or stakeholders who may need additional information.
You can see this (fairly old by now) quote from Glassdoor below how one engineer saw value in this process, writing, “The transparency principle gave the feeling that everyone is involved.” This level of transparency is so important in raising the level of consensus and commitment to our architecture.
We acknowledged our mistakes and incorporated our learnings.
It’s crucial to process negative feedback in a way that makes people more aligned rather than driving them apart (this is important so we’ll return to this point a little later).
I’ll add that a key part of architecture is to get consensus, and to do that you need to build understanding. This involves sitting down with key stakeholders and talking them through it, though there are many ways to go about this. You can do a group presentation and record it. You can walk them through what is being proposed if they are too busy to read the RFC. Be prepared to hustle to get people’s attention and to let them know that what is being proposed will affect them. All this is a key part of the socialization aspect of architecture.
Step 4: We worked hard to get buy-in and authentic feedback
Finally, it was not just about sharing the work we were doing; we also needed to get others bought in and comfortable sharing their feedback with us. This involved creating psychological safety around negative feedback. The last thing I want is a junior engineer being fearful of sharing negative concerns about our architecture, or sharing the feedback and getting shut down. That’s a massive anti-pattern.
When I re-architected the League of Legends game client at Riot Games (now on over 150m desktops), we fell into the trap of sidelining people with negative feedback. This was a big mistake, as they were trying to tell us valuable information. It took us a while to recover from that and establish a more psychologically safe environment. Riot Games has a strong feedback culture, but learning to take feedback constructively is a core skill to learn.
So, a key guideline here is to separate the emotion from the actionable feedback. When someone offers feedback, it’s important not to take it personally. Listen to what they’re sharing, and ask for more information if necessary, but try to approach it with a curious mindset rather than an emotional one.
As a quick recap, the four steps involved in setting a technical direction at Suvoda were:
Getting clarity on the product direction
Creating an ambitious North Star and devising a conservative, incremental way to get there
Communicating transparently
Working hard to get buy-in and authentic feedback
Mistakes and pitfalls to avoid
Now that we’ve seen what happens when you do things right, let’s look at a few of the mistakes, pitfalls, and anti-patterns to avoid.
Mistake #1: The big rewrite
I was involved in a very large project in 2002 at a financial institute. A consultancy came in and said they should rewrite everything into one system where core financial products were modeled as a series of cash flows. This led to a division between the legacy team and the next-generation team.
Having this type of division is a disaster waiting to happen. The legacy systems were delivering immense value and had done so for the company’s history. Asking the team to rewrite everything while running parallel development made it hard to move people onto it.
In this particular case, I saw what happens when everything that could potentially go wrong does. The company declared the project a total failure and had to destroy its source code to treat it as a tax write-off.
In many cases, a big rewrite is just a way of avoiding understanding what the current system really does. It’s harder to evolve and read code than it is to write it, which is why a lot of people avoid it. And you can keep delaying for a while, but eventually, you realize that the new version isn’t even close to the capability of the old system.
Another famous example that happened years earlier was Netscape deciding to spend three years rewriting the core of its browser. While they worked on the new version, they helplessly watched as their market share plummeted. Joel Spolsky calls those big rewrites “the single worst strategic mistake that any software company can make.”
I’m not saying that you can’t do rewrites—every now and again you will have to. But my advice is to try to scope it down to the smallest possible thing to rewrite. You want it to be in a bounded context. There’s always value in existing systems. It’s a massive anti-pattern to devalue the teams that got you to this point and then try to completely re-engineer. Do this as a last resort.
Mistake #2: Not delivering incremental gains
When you rewrite software, the journey has to be worth more than the destination: not delivering incremental gains to the business is a fatal mistake. You’ll never know where you’ll end up and when you’ll be there. And to be frank, the chances that the destination you reach is the one you originally planned are thin. North Stars are incredibly useful to set direction, but be open to the possibility that the destination will change as you find value along the journey.
You’ll always be reassessing your end goal as you progress. If you don’t provide value along the way, at some point, someone will come along and say they’re not going to wait three years for you to finish this project and that they need to see benefits now.
At Suvoda, we always make sure each step we take towards our North Star brings business value—see step #2 above where I described how to create an ambitious North Star and devise a conservative way to get there. Case in point: one big incremental gain we implemented at Suvoda was “templates” for organizations. We quickly realized how powerful they were to the point that we decided to update our North Star accordingly.
Not taking this approach runs a very high risk that someone will come and cancel your plans. Define an end goal, reassess frequently, and deliver incremental gains all along the journey to avoid this mistake.
Mistake #3: The big vision that can’t be achieved
I’ll quote Steve Jobs: “Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains.”
Often, your company will have a vision that’s unclear to your teams. Either they don’t know what it means or they don’t know how to achieve it, because it’s too complex and over-engineered. This can be related to what Fred Brooks calls the second-system effect: when V1 is successful, we tend to be overconfident and overcomplicate V2, creating more chaos than necessary.
When teams understand the main goal and know it’s achievable, they’ll work hard toward it as they’ll feel it’s within grasping distance. But in the opposite case, they’ll give up and set the company up for a Death March: a project with pressure so intense that teams will only strive for survival. This is what happened at the financial institution I worked for mentioned earlier.
Set simple, clear, and achievable goals to avoid both anti-patterns.
Mistake #4: Not solving the right problems
There are always plenty of areas where we can focus our attention, and sometimes we might choose to address issues that make us feel good without really tackling the core problem. The visual on this slide helps illustrate this point, with the large red bubble representing the core issue and the smaller white bubbles representing related, but less important issues.
In one case at LiveRamp when I was Chief Architect, we didn’t look enough at how the big data processing pipeline was working (our red dot in that situation). This happens a lot when there’s a large monolith with a lot of business logic in it. People stay away from the monolith and look for other problems to solve. But here’s an important lesson: you’ve got to attack the heart of the problem, even if it’s hard, and take the incremental steps towards the North Star. Otherwise, you’re just delaying the inevitable.
Mistake #5: The hype trap
Every five years or so, there’s another hype train.
I saw this 20 years ago with Enterprise Java Beans (EJB), 10 years ago with Single-Page Applications (SPA), and at the moment with microservices. While there’s nothing wrong with a new technology in itself, the problem is jumping to adopt it without identifying a real benefit for your business.
Let me share an example that went wrong at a previous company. We hired 15 people to rebuild our UI as an SPA. A few weeks in, the C-Suite stepped in to understand why we were focusing on rebuilding a new UI. Incapable of justifying the business value, the main argument in favor became “We must rebuild because we brought in 15 people” - the sunk cost fallacy. We eventually shut down the initiative and it took months (+ drama) to transition these 15 individuals who then went on to deliver massive value (and revenue) on another project. As an aside, not one client had complained about our UI in the first place, and when we surveyed them, many preferred it over our competitors.
This is another example of when people tend to avoid hard questions and follow trends without critically assessing whether they make sense for their own company and context. You always need a valid business reason to do anything, no matter how shiny and new it may appear at the time.
Mistake #6: Not getting allies and buy-in
Sometimes, office politics can put a whole project at risk. I’ve seen this happen with an immensely popular product at Riot Games for the league client update. People were sniping at us and complaining to management rather than helping us to do it.
And this ties into another problem—the inability to process negative feedback productively. People were having emotional reactions to negative feedback, considering it “bad-mouthing.” This undermined the many valid points the critics were making.
It wasn’t until we started listening to people’s relevant objections in a more rational way that we started moving the needle. Accepting negative feedback without an emotional response is one of the most effective ways to gain allies and buy-in. People want to see that you’re taking their insights and expertise into account.
This is why in such cases you need to process everyone’s feedback—no matter how they formulated it. Ask yourself: “Are they right?” It’s okay to consider it and disagree; it’s also okay to dislike the way feedback was shared yet acknowledge the point.
When you do that, people will champion your cause, and it will start percolating through the whole company. Get allies and buy-in by putting office politics aside and listening to feedback.
Mistake #7: Not managing upwards
As we are managing projects, we mostly think of how we should organize our team, tasks, and goals. But we often forget about managing upwards. A lot of things can go wrong if we don’t—the CEO and other executives can get disconnected and ask you to go in a different direction. If you don’t make sure your own managers understand the value of your project, they can end up killing it or worse, make you work on the wrong items, leading to a disaster.
This is especially damaging if they don’t understand the challenges at stake in the process, don’t get learnings from it, and aren’t necessarily capable of carrying off the project. When the people who are in charge don’t have the insights, it creates engineering inefficiency.
You have to continually be a self-promoter when you’re working in a company. Other people won’t be aware of what you’re doing unless you tell them. This is challenging for engineers because they believe their work should speak for itself.
Another aspect as architects is that we have a key role in surfacing the ground truth of engineering to senior management. We should be in the code, seeing what is working and what isn’t, and this leads to far more informed and educated decisions. Imagine how successful development can become if management operates based on a true understanding of their systems!
Wrap-up and final thoughts
Now you’ve seen what works well when setting a technical direction—and what doesn’t. I hope you’re able to learn from these mistakes so you don’t have to repeat them!
To sum up, here are my key takeaways:
Every company already has the core knowledge and expertise to create a successful technical direction that is achievable.
A good direction allows value increments, conservatively, without sacrificing the speed of delivery. Tailor this approach to the company you work at, and the industry norms.
Getting alignment is a valuable skill to develop (and it will likely take some time and experimenting with different tactics to get it right).
Processing negative (and positive) feedback is a learned skill. Your goal is to take it in and decide which actions to take without letting your emotions control your reaction.
Finally, my core message: Don’t delay answering the difficult questions when you re-engineer your systems. This is just delaying and amplifying the eventual pain.
And that’s a wrap for today!