Satisfying an external auditor when using Agile

Question

My company is an ISV and occasionally, prospective customers (especially in life sciences) insist on sending in auditors to check over our software development processes.

We are starting to implement Scrum/Agile and my CEO is very concerned about how such an auditor would react to this. His concern (which I share to some extent) is that an auditor would expect to see a signed-off specification document against which the software is developed and then tested, and they wouldn't be happy with an incremental approach which doesn't really deliver a full specification at all.

Does anyone have any experience in this respect? Is it usually necessary to fudge the issue slightly by "signing off" a user story, or producing a retrospective specification document etc.? My only defense to the CEO so far is that agile is almost universal in software development, so they are bound to accept it as valid, but I don't know whether this is true.

I have seen some discussion about ISO 9000 and the general consensus seems to be that agile is not incompatible with that, but I don't think that's really the same question because my sole motivation is to be sure we can satisfy an auditor. FWIW We are not ISO 9000 certified and have no intention of doing so, and auditors have not had a problem with this in the past.

Answer 1

Agile development methods are not "don't specify up-front; don't document" methods. That's called chaos. When approaching an agile development project, you start by capturing the requirements as a set of user stories, which are added to the backlog. That backlog, along with a documented acceptance criteria, forms the specification. If you need a physical signature, print them off, sign them and stamp them as the master copy. That may be over the top for many auditors though: they should be happy with those stories being help in an issue/ticket system that requires password access, in a document management system, marked as approved etc.

During each sprint, stories are taken from the backlog, potentially decomposed into tasks, and worked on. Again, you need to record (on paper or in a password-protected system) each of those actions (eg, formally record somewhere that a task is done, even if that's just closing the task in eg Jira). That way you build up an audit trail of both any changes to the specification and the work done against it.

Finally, document what you do as a set of procedures and ensure all staff are aware of those procedures. One word of warning: always document what you do, never document what you'd like to do, for that road leads to endless audit problems.

One final thought: the easiest path to ensuring your auditor is happy is to discuss your plans with them at their next visit. Have your processes documented, shown them to the auditor and explain how it will all work. If they aren't happy, they can then guide you in what else you need to do.

Answer 2

The solution to your question is in "change management". What is important for the auditor is not that what you decide when you start a long project is valid at the end of the project, but that all changes are linked to decisions taken by the right people and that these are documented.

So you will have to document your global plan (not always very Agile, but necessary to run the company anyhow), and then document your decision meetings (Scrum meetings are very good because these include the customer (or its representative) and the doers). You will also have to document your acceptance criteria (test suites ...) and document that your product are released according to your acceptance criteria.

If you keep this documentation up to date and you really include the product owner in the scrum meeting (he is representing the customer), then this part of your quality system should be fine with the auditor.

Answer 3

Check whether your prospective customers are in a regulated environment

I successfully managed a very stringent audit of this type, though for a different vertical. We handled very sensitive financial data and we had to prove to the auditor that we had the necessary safeguards in place. As @Marv Mills said the auditors focused on making sure that we were following our stated process. However, we were not Agile then.

Here is a good article - Being agile while still being compliant A practical approach for medical device manufacturers

In summary:

"Many organizations believe that they must use a waterfall approach in regulated environments," they say. "We show how this is not the case."

So, I would say that the first thing for you to do is to find out whehter your prospective customers are in a regulated environment. If the answer is yes, find out if the regulations are prescriptive or descriptive.

In the prescriptive approach, the regulations define what must be done and, in many cases, how it is to be done. In the descriptive approach, the regulations define what must be achieved and largely leave the question of how to achieve it to the regulated organization.

If they are in a descriptive regulated environment take the following steps:

Map regulated activities into your agile development process.

Insert "hardening" sprints at certain points during development as synchronization points to ensure that the software meets all regulatory requirements.

Answer 4

From what I've read previously an ISO9001 auditor is likely going to do the following things:

Inquire as to what your engineers are doing
Ask them how they've established that this is actually what they should be doing
Ask to see the project documentation which should match

Agile has plenty of documentation, so this should be easy. Although it's not the same documentation as a Waterfall environment, it will be there. You should have your processes documented on a wiki or in project documentation available to the grunts, but they should easily digestable (simple/lightweight). That should be enough for an auditor.

Answer 5

As an IT auditor, i would expect that there be guidance (policies and procedures) for the process you are following, that you can describe the process you are following, and that you can prove you are following the process. Step 1 of the audit is to make sure that the documented process (policies and procedures) are sufficient to mitigate risk. If step 1 passes the audit, then i ask for proof that the process was followed.

What i have seen with Agile is that there seems to be a false perception that documentation is a waste of time that hinders the creative flow of the agile process.

Answer 6

Yes its true that for functional requirements the user stories and acceptance criteria (and Definition of Done) provide sufficient auditing cover for most purposes. But for non-functional requirements the situation is different, especially those that are compliance related.

Non-functional requirements like "meets level of security", "uses only components" won't show up in a user story's Acceptance Criteria, because they would need to be in every story and that's not practical. Usually they appear in Definition of Done, which in compliance environments can get fairly large.

The audit question then is whether your team process can demonstrate that for every story Accepted, the Definition of Done items were validated line-by-line. In a high-functioning trust environment training and occasional inspection may be sufficient to confirm. In tighter or less trustful environments, a checklist might be necessary (and that could be attached to every story).

Answer 7

I haven’t had experience with external auditors but gained plenty of experience dealing with internal Business Owners that fund projects, who - 1. have little experience with Agile methodologies and practices (which is alright because all of us were new to it at some point), 2. were uncomfortable adopting new practices (which makes it tough for the dev teams who intend to be Agile).

First thing I would do if I were you, is know the audience (auditors/customer in this case). I would try to gain more insight into the customer’s history if possible. Given that Agile practices in software development have been around for quite some time, chances are the customer is already used to it. If I got to know that they are used to it, I would show off the practices that we already follow.

In case I couldn’t gain such insight, or if I got to know that they never practiced agile, I can directly use the learnings from one of my old projects... There we had to do two things: 1. We had to prepare a roadmap from the Backlog by adding timelines against each line item, just so that they feel comfortable looking at what they see (because that’s what they were used to). And we highlighted the confidence level on meeting those dates (example: “70% confidence that we will meet this date”). 2. Added granular requirements to each of the stories (even though we knew that those will most likely change) - again to make them feel comfortable

However, our execution continued being Agile. Presentation was modified a little bit (though not wrongly modified at all) so that the audience understood it in the traditional language.

Answer 8

I commented on David Arno's answer, but worth one of my own.

The Product Owner controls your backlog priority.

As part of story creation you create the acceptance criteria (usually in the form of questions)

This is reviewed by the product owner (or the business sign off proxy), this is your spec, and it should be agreed before the story is worked on).

You may want to document your sign off (or change the workflow in your tracking tool to record it).

Agile doesn't mean we abandon everything, we do "just enough", so the criteria is our spec.