Home / Growth Culture / The importance of testing velocity (not speed) in growth

The importance of testing velocity (not speed) in growth

Article originally published in February 2023 by Stuart Brameld. Most recent update in April 2024.

Request a demo

Project management for growth and agile marketing professionals. Map your acquisition funnel, integrate analytics and run agile experiments.

Experiment results

Recent experiments results include competitor SEO, AI-driven content, exit-intent modals and AB testing homepage headlines.

Case study

"We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results." 

Certified

We are vetted mentors with Growth Mentor and a partner with the Agile Marketing Alliance.

An agile, iterative approach to marketing is at the centre of many leading growth marketing teams. We have talked about the growth marketing process and the benefits in building a culture of experimentation before.

One of the best ways to measure growth marketing success is to shift the focus from monitoring outputs (traffic and leads) to monitoring inputs (experiment speed and quality).

Inputs are what you actually control.

Andrew Chen

Why testing velocity matters

When Satya Patel joined Twitter as VP of Product in 2010 one of the things he insisted was that the Twitter team move from roughly one test every other week to 10 tests per week. Twitter grew rapidly between 2010 and 2012 and it is widely believed this had a lot to do with an exponential increase in testing velocity (from 0.5 tests per week, to 10 tests per week).

source

Similar, Sean Ellis describes how growthhackers.com hit a plateau upon reaching 90,000 active users after the first year of launch. Without any increase in budget or changes to the team, they were able to grow from 90,000 active users to 152,000 active users in 11 weeks by dedicating themselves to high velocity testing.

Source

“Move fast and break things. Unless you are breaking stuff, you are not moving fast enough.”

Mark Zuckerberg

Geoff Charles, VP of Product at Ramp, describes how the world’s fastest growing SaaS company approaches work:

“In essence, we believe that doing is better than planningAny second you spend planning is a second you don’t spend doing. The moment you are aligned in a direction, you don’t need a high level of accuracy. It’s impossible and costly to try to predict what you can do—and part of our competitive advantage has been that we can respond very quickly to the change in environment, strategy, or customer feedback. We learn something new every day that helps us adjust our plan.”

Geoff Charles, VP of Product, Ramp

Monitoring testing velocity

There are a number of metrics and KPIs available to monitoring a growth marketing, experimentation or conversion rate optimisation programme. Essentially these boil own to 2 areas:

  1. Increase tests run (quantity)
  2. Increase tests won (quality)
Want to knowMetricArea
How many tests can I run?Testing capacityQuantity
How many tests am I running?Testing velocity, testing coverageQuantity
Am I getting better?Trends over timeQuantity
Am I running effective tests?Win rate, lift amount, expected valueQuality
Am I running tests effectively?ROIQuality
Am I getting better?Trends over timeQuality
Adapted from slides by Claire Vo

Testing velocity is about the number of experiments run in a particular time period and tracking this trend over time, towards a goal based on your testing or experimentation capacity.

For example, assume your average test takes a month to research and start and you have a team of 5 growth marketers:

  • 1 experiment per person per month = 5 experiments per month
  • 1 experiment per person per week = 20 per month

The difference between speed and velocity?

Speed and Velocity are both used when describing motion, and are often used interchangeably, but which have very different meanings.

“There is a difference between “speed” and “velocity”. Speed is how fast something is going, while velocity is how fast something is going in a certain direction.”

Lukas Vermeer

The official definition of velocity from physics is “the rate at which an object changes its position”. Unlike velocity, speed has no direction, and it simply describes how fast an object moves regardless of direction.

This is where many teams fall foul of delivering outputs over outcomes. Speed focuses on work/activities/things done, without any considering for value delivered (to the organisation, or to the customer). Velocity measures both the quantity of work done and it’s quality through ensuring alignment in a particular direction (typically Traffic, Leads, MQLs etc).

Image source

As growth marketers, we want to be focused on testing velocity. You can consider this as speed aligned with a specific business outcome or goal.

The more tests you run, the more you learn about how to grow your business. So it’s only natural to want to run as many tests per period of time as possible.

via GrowthHackers

Agile velocity

Agile velocity is extremely popular and when known within the software development world. It is the average amount of work a software team completes during a software development iteration or sprint cycle of a given timeframe.

As above, software development teams want to release new features and functionality quickly (i.e. speed) but must also release features that add value (i.e. velocity). Customers don’t care about tickets completed, or features shipped. Similarly, they don’t care about how many articles you have on your blog, or the number of marketing campaigns you run every year.

Speed matters, but we also need to ensure that we are speeding in the right direction. Running into a wall faster is not a recipe for product success.

Velocity makes it easy for agile teams to estimate how much work they can achieve per sprint and how long it’ll take to get a project to a certain level of growth. It is however a metric that is calculated retrospectively (how many backlog items were delivered in the sprint?) hence is not a good way to budget or forecast.

Monitoring testing velocity

Testing velocity is a key metric we monitor in Growth Method and should be a key metric within any marketing project management software in order to ensure delivery of outcomes over outputs.

“Smart leaders realise that the key is not in doing a lot, fast, but in doing the right things, and that’s where they spend most of their energy—helping the organisation make good decisions.”

https://itamargilad.com/velocity-vs-impact/
Image source

We believe testing velocity should be a key considering in your marketing strategy, and marketing operations.

Key takeaways on testing velocity

  • Doing is better than planning
  • Time spent planning is time not spent doing
  • An increase in velocity = an increase in learning
  • You don’t need a high level of accuracy early on
  • It’s impossible and costly to predict what you can do
  • Use learnings to continually adjust and course correct
  • Remove as many dependencies as possible

Got questions? Ping me on LinkedIn or on Twitter.

Here are some additional resources and articles on testing velocity you may find helpful.

Here are some related articles and further reading you may find helpful.