Moving to an agile ‘in-flight’ assessment model

DTO transformation projects are now being assessed more regularly using an ‘in-flight’ model. This gives teams a chance to iterate based on regular feedback and work in a more agile way. DTO’s Head of the Digital Service Standard, Jacqui van Teulingen, explains.

Image: A project team engaging with the assessors using an agile ‘in-flight’ model.

Up until recently, the DTO has used an assessment process we adopted from the UK’s Government Digital Service where we did assessments at the end of each stage as part of the service design and delivery process.

We saw missed opportunities

Having now assessed 4 DTO digital transformation projects this way, we have learned that the process could be better. Why? Assessing at the end of a stage as a once-off, didn’t allow us to support teams in applying the Digital Service Standard in the way we had anticipated. Providing recommendations at the end meant that we were less able to keep the product on track. Plus, it was not an agile way to assess an agile product.

Teams were putting time and energy in the wrong place

Service teams were spending a week preparing presentations and material for their assessments. Many teams approached the assessment like a test they had to pass. It meant that there was a lot of stress and a huge reliance on good storytelling. While we avoided using the language of an examination, the set nature of a pass/not pass at 1 single point made it feel like this.

Assessors weren’t feeling confident

Assessors had to become familiar with the service and what it had achieved in 3 hours. They could never be sure they’d asked all the right questions. It sometimes came down to whether the team told a good story. “I felt like the real estate agent coming in for the annual rental inspection, right after everyone had just cleaned up. What was happening between inspections?” said one assessor.

A more agile (and fairer) approach

In response, we’ve iterated the process and have moved all the transformation teams to the new ‘in-flight’ model. This approach focuses on regular check-ins by assessors with the project teams.

Assessors, who are all experts in their field, are assigned to the project at the beginning. They become familiar with the service’s intent, the evolution of the product and the team’s way of working.

A chance to improve before it’s too late

In the new assessment model there are weekly check-ins, and in consultation with the team, the team is now rated against the Standard using a Red/Amber/Green status. Assessors provide recommendations ‘in-flight’ when the team is able to respond, not at the end when it’s too late.

A mentoring relationship gets built

The in-flight model establishes a mentoring relationship between assessors and teams. The team and assessors determine when they are ready to transition from one stage to the next. The transition points become celebrations for the team and the assessors. At the transition points, the DTO will now publish a short report on the product’s progress and the lessons learnt.

A more collaborative way to achieve a good end result

Assessments against the Digital Service Standard were never about passing or failing anything - they were about making a better thing. This new ongoing and interactive approach is all about working collaboratively to do exactly this.

It may take a little more commitment and time on the part of the assessors, but feedback so far from the teams include:

‘[The new model] really helped the team show and tell the awesome work we’re doing.’

‘We’re all loving the new format and the great advice from the assessment team’.

What happens next?

We will soon add information and advice to our Meeting the Standard page. If you prefer the old ‘staged’ assessment method, or are in the process of applying it, that’s okay. This is still available as an option.

We’d like to know which method works for you. Have you had any experiences using the staged method that you can share with us? What has worked? What’s been a challenge?

Stay tuned for our first in-flight assessment report which we will share soon.

Want to join the conversation?

Read our comment moderation guidelines