Organizations certified as ISO 9001 compliant need to conduct Internal Audits of their processes at planned intervals. The objective is to determine whether the processes followed in their projects are effective and are in compliance to the requirements defined in the organization’s QMS (Quality Management System).
Auditing a project implementing agile processes has some unique challenges. Such projects do not perceive any direct value addition from these audits to the product they deliver to the customer (unless the customer has mandated such an audit). However from the organization’s perspective an effective internal audit is a means for continuous improvement.
Here are some guidelines for auditing a project which follows Scrum an agile methodology:
- Audit should be non-intrusive as far as possible
- Audit should not trigger creation of for-Auditor-only documents
- A generic Scrum checklist tailored to suit project requirements should be used as the basis for audit
- An Auditor is assigned to an entire Sprint as per the Internal Audit Plan
- Auditor is a silent observer of the Sprint.
- The Auditor is added to the team mailing list to receive all communications; provided access to all the artifacts; attends at the minimum Sprint Planning, a few Daily Scrum meetings, Sprint Review and Sprint Retrospective meetings.
- Auditor does not schedule formal audit meetings with the team members but may seek clarifications from ScrumMaster and/or Product Owner as needed during the Sprint.
- Auditors prepare the audit report recording their observations and findings against the items in the checklist. However they are encouraged to go beyond the checklist and provide their suggestions for improvement.
- The Audit Report is presented to the Team preferably immediately after the Sprint Retrospective meeting.
- Non-conformances are addressed in the forthcoming Sprints and verified by the Auditor.
The Internal Audit following the above guidelines will minimize the project overheads, allow the team to be focused on delivery and yet can yield value added recommendations from the Auditor.
[Please feel free to leave your comments below or bookmark/share this post]
Would the Auditor be open to feedback from the team on how the auditor may have interpreted particular events within the sprint and captured the auditor’s view in the report? I ask because of the statement: “Auditor does not schedule formal audit meetings with the team members but may seek clarifications from ScrumMaster and/or Product Owner as needed during the Sprint.” Which tends to mean the auditor is ONLY an observer and should not interact to seek context and full understanding. Perhaps I’m misreading the intent. If I were an auditor – should I ask for understanding?
Hello David,
Good question.
Yes an Auditor has to be open to feedback from the team. The report is finalized only after taking into account the team’s inputs . The non-conformances to be addressed are mutually agreed upon.
The only reason why I stated that “Auditor does not schedule formal audit meetings with the team members…” is because it will distract the team from the Sprint work.
If the Team while planning the Sprint, factors in the time likely to be spent in audit meetings, then it is fine to have such meetings. However in such case Sprint velocity may be somewhat lower than what the Team normally achieves.
Gopinath
Hi,
Can you send me the checklist you might have used to do audit of a agile project . Thanks in advance
Thanks
Mayur Sand
Mayur,
Checklists are very specific to a project or an organization. You need to evolve one which suits your context. However you can use the checklists available at the following locations as a starting point:
http://www.crisp.se/scrum/checklist
http://www.scrummasterchecklist.org/pdf/scrummaster_checklist09.pdfhttp://scrummasterchecklist.org/pdf/ScrumMaster_Checklist_12_unbranded.pdf
Hope this helps.
The checklists you mention only cover Scrum. How do you audit ISO compliance? ISO 9001 Clause 7.2.2 states that where the customer provides no documented statement of requirements, the customer requirements shall be confirmed by the organisation before acceptance. In Scrum this may be a challenge. assuming the product backlog is complete and approved by the client (which is not given) then if only user stories are given and no formal requirements exist the team may not comply.
Further, clause 7.3.1 says that the organisation shall determine the design and development stages. This can probably be mitigated, but how do you audit it?
Other clauses which are challenging includes 7.3.3, 7.3.4, 7.3.5, 7.3.7, 8.1, 8.2.3, 8.2.4, 8.5.2, 8.5,3. I don’t think the checklists you have suggested cover any of these clauses. However I so think these clauses can be satisfied but it may demand some tailoring of the basic scrum process. How have you audited these aspects?
@Mikael: Thanks for your comments.
I agree that the checklists cover only Scrum and the compliance with ISO 9001 is not very explicit in it.
But they cover all the ISO requirements in spirit . To reconfirm I read through the ISO 9001 standard , all the clauses you have mentioned and I did not find anything which is a blocker to agile implementation.
Any reasonable and pragmatic ISO auditor will not raise issues if you are following Agile in true spirit.
Yes you may face some problems if the auditors start asking for formal signed-off documents. This usually happens when the auditors do not interpret ISO correctly and also do not understand the process of Agile development. But it will be good in the longer run if you could take up the challenge of convincing the auditor that the practices you are following meets the intent of ISO.
Agile per se is not against documentation. It advocates writing documents that makes sense, to the details it makes sense (i.e. don’t over document) and to the formality it makes sense to the business.
Your questions are very relevant and have prompted me to think about writing a blogpost on how to map ISO & Agile practices. Hopefully I will do that very soon.
Well I was not so much thinking of signed documents as documented evidence. I.e. ISO actually demands evidence. This can be done by photos of the white board.
In additions user-stories alone may not satisfy ISO but you may need to consider adding acceptance criteria or some other less ambiguous means to satisfy the ISO requirements.
There are a few papers discussing this in particular Stålhane & Hansen’s excellent paper – The application of ISO9001 to agile software development. Product-Focused Software Process Improvement 9th International Conference, PROFES 2008, LNCS, Vol 5089, pp. 371-385. Springer, Heidelberg (2009)
Mikael, I agree that the statement of user-stories alone may not satisfy ISO. You may be aware that the purpose of the user story is to trigger, Conversation & Condition of Acceptance. Where necessary it also needs to be supported by artifacts like wireframes, screenshots , architectural diagrams. Besides an ideal user-story needs to satisfy INVEST (Independent, Negotiable, Valuable, Estimable, Sized appropriately and Testable). This combined with several good engineering practices (mainly from XP) like TDD, Continuous Integration, Refactoring, Unit Testing, Code reviews etc. will meet most of the ISO requirements. Acceptance Test Driven Development is another new trend which is catching up where the requirements are written in form of acceptance tests.
Thanks for the reference to the Stålhane & Hansen’s paper. I have downloaded it from the net and will go through it.
Totally agree with you. i am also a Scrum Trainer & i know Your hard work.
Thanks Faisal.
It would be nice if you could share some additional viewpoints or your experiences in auditing Agile projects.
I appreciate this list, as an agile practice head I always see lot of challenges in auditing distributed agile teams, esp if the onshore team is completely in diff timezone.
Thanks Shankar.
Would like to hear more about your experience in auditing distributed agile teams
This is very interesting. How do we see this guideline in the view of CMMI Maturiy Level 5 compliance?