Why are there so many Agile Coaches?

Makes me wonder.

Agile development must be hard. Why else would there be so many Agile coaches? When people were doing waterfall, I don't think there were that many "Waterfall Coaches". Think about it. How many of your contacts in the software world were "Waterfall Coaches"? None of my personal contacts were coaching waterfall. On the other hand, I know of at least half a dozen who are in this business - either fulltime or part time. And I'm running into many strangers who are Agile coaches. Or at least want to become one.

For a methodology that is supposed to be simple, why are there so many coaches? Perhaps it is not simple. Agile is supposed to simplify but that's not the same thing as being simple. Perhaps the highly adaptive nature of agile methodologies necessitates a coach.

Essentially - what does an agile coach do? If you are a team that's already practicing an agile methodology, she'll probably take a look at your methods and suggest improvements. Should I use the word prescribe changes? Funny. We choose a methodology that's not prescriptive for its ostensible benefits and then we hire a coach to prescribe methods.

The Rise of the BAQA ("BaaKwaa")

The role of the Business Analyst (BA) and Quality Assurance Engineer (QA) as we know it is changing. As agile methodologies become prevalent, they appear to have an overlap. I think of a person performing this new role as a BAQA (pronounced BaaKwaa). The need for people in agile teams to wear different hats is spurring this development. In recent times, this has received a boost with the development of frameworks supporting Behavior Driven Development (BDD) and Acceptance Test Driven Development (ATDD).

Agile teams, and in particular teams following BDD/ATDD, write user stories in place of verbose software requirements. The user story captures the essence of the requirement expressed in a "Role/Feature/Benefit" grammar. Subsequently, acceptance criteria are written as "Givens/Events/Outcomes" to exemplify the requirement. Together, they completely define the business requirement.While the user story requires a Business Analyst's skill, the acceptance criteria requires a Quality Assurance Engineer's skill. Essentially, we now have a need for a person who can play both roles.

Enter the BaaKwaa.

Let us see the typical responsibilities of a Business Analyst and QA Engineer in a traditional methodology, such as waterfall.

A Business Analyst:

• Understands the business domain
• Analyzes weaknesses and strengths and helps define new features
• Writes software requirements specifications for the development team

A Quality Analyst:

• Writes test cases
• Performs tests on software programs and finds defects
• Ensures the quality of the software

Here's a brief role description for a BaaKwaa in an agile team:

• Write the user story (thus capturing business requirements)
• Write the acceptance criteria (thus further expressing the requirements through examples)
• Write the acceptance criteria in a BDD/ATDD framework in the language of the framework (example: Fitnesses, Cucumber, StoryQ, etc)
• Verify that the implementation meets the acceptance criteria preferably through automated means.

I'm not suggesting that we are witnessing the demise of the specialist BA and QA. There is a role for the specialist and there always will be, just as we continue to have architects, DBA's and user interface specialists in agile teams. However, they will be required to drop their BA or QA hat quite often, and instead wear the BAQA hat.

Please welcome the BaaKwaa.

Agile: The Preacher and the Practitioner

In an Agile conference, you will find two types of attendees - the preachers and the practitioners. The preachers are the speakers, the trainers and the agile "experts". They believe implementing an agile methodology in a text book like fashion is the only way to do it. The practitioners are those who have started an agile process in their company, are struggling to make it work and are dealing with real organizational and people issues.They see value in the agile process but do not think it can be done in a textbook fashion in their organization.

Here is what, for example, some of the agile preachers believe:

• SCRUM-But is evil. Not implementing an aspect of SCRUM points to an impediment that must be fixed.
• You must have teams co-located.
• Autotmated testing (TDD), continuous integration, etc. is a must.
• The Product Owner is always available to answer the team's questions. 
• Team resources are fully dedicated to the sprint. 
• Deadlines can and should be stretched to acommodate quality.

Here is what, for example, some of the agile practitioners have experienced:

• SCRUM-But is a reality for organizations dealing with legacy products.
• Co-location is a luxury in today's era. Often, even in the same geographic location, some team members work from home.
• Automated testing, continuous integration is often not possible for legacy applications.
• The Product Owner (Product Manager in the real world) is often managing multiple projects. It is an illusion to assume the Product Owner is dedicated and available.
• Team resources - especially QA, DBA, Architects often work on multiple projects or agile teams concurrently.
• Deadlines are real. They are often final and products must meet them and make compromises in the process.

The Agile Preacher often comes off as a white collar intellectual mostly used to working in laboratories, where things are largely under control. The Agile Practitioner, comes off as a blue collar worker, just interested in getting the job done in the real world with real people and real challenges to face.

The truth, as usual, is in between. Agile Preachers must temper their language to factor reality. Agile Practitioners must try to change reality. And in this tension between the two groups is where we'll see real value being added to the Agile Process.

Agile Boston Open Space 2010

I attended Agile Boston Open Space 2010 (http://www.newtechusa.com/agileboston/events/AgileBostonOpen2010.asp) today. The event was at the Microsoft Building in Waltham, Boston. Of the many conferences that I have attended, I have no hesitation in saying this gave me the most value for money. Here's why:

1. Convenience: It was in my backyard - just a couple of miles from my workspot. It certainly helped that while the title had Boston in it, it wasn't in Boston. It was in Waltham (you can tell I don't like going to Boston).
2. Cost: Just $29. And that included breakfast and food. And yes - there was vegetarian food. It was just a wrap, but it was tasty. This is not a trivial matter. I'm a vegetarian and I have often found that some conferences that charge heavy fees don't pay much attention to the vegetarian fare. These people were thoughtful.
3. I got to hear Ken Schwaber for an hour.
4. I got to network with a number of smart people who were willing to teach, learn and share. I learn't about a number of misconceptions about Scrum. Just because it is highly adaptive doesn't mean that you shouldn't do some of the process that truly added value in waterfall. People didn't say that in so many words, but that was the gist of what I got from it. If it's valuable, keep it.
5. I learn't about "Open Space Technology" (http://www.openspaceworld.com/brief_history.htm) - a fabulous way to organize meetings very productively.

I think it was a day very well spent.

TDD is a Misnomer

We all know, at least those of us in the software community, that TDD stands for "Test Driven Development". That's a huge misnomer. Simply because TDD is much more than about tests. Yes - tests are a core part of TDD. But TDD is also about the following:

1. It's a way to capture behavior (think requirements)
2. It's a design tool (If it's easy to test, the resulting code will be elegant)
3. It captures high level documentation (think tools like agiledox)
4. It's a regression test suite.

The word "tests" in it leads to all kinds of confusion among TDD practitioners. What do I test? How do I test? What do I test first?

Happily, we now have Behavior Driven Development - which simply removes the confusion around TDD while taking it to a whole new level.

Yes - I'm beginning to like BDD a lot. It's a better TDD and more.

ScrumBut is not that bad

ScrumBut is a practice where teams implement partial apsects of Scrum. As in - "We did Scrum BUT we did not have daily standups". Scrum pracitioners look down on teams that practice ScrumBut and perhaps for good reason. As they argue, when a team does not implement a prescribed practice in Scrum, the team is allowing for a problem to fester.

For example, if a team finds that a Product Manager is unable/unwilling to participate in an agile team as a "Product Owner", using ScrumBut, the Business System Analyst may take on that role. The "Product Owner" is one of three key roles that Scrum prescribes. Not having the real Product Owner play that role is a violation of Scrum. In reality, there might be organizational hurdles, as in highly matrixed organizations, that prevent the Product Manager from participating in agile teams. Should the agile team give up their move towards adopting agile methodologies?

Perhaps not. ScrumBut is an excellent way to get started with agile methodologies. Agreed that it is not likely to give the same amount of return that Scrum promises. However, it is an incremental way to adopting Scrum. It is likely to result in less of a culture shock to new teams and may provide the breathing space to winning hearts and minds.

The end goal though must be to eliminate the "But" and have just "Scrum". Assuming of course, Scrum is what you want for an agile methodology.

Embrace “Automated Unit Testing” in Waterfall

Automated Unit Testing is a process where developers programmatically test their code using test frameworks such as the XUnit framework. Code is not considered complete unless it is accompanied by a test case. Agile teams have embraced unit testing. Waterfall teams can also do so without disruption to their process.

Introduction
Agile software development teams have reaped significant benefits by adopting “Automated Unit Testing” as a part of the development process. For teams that are considering becoming more agile, adopting “Automated Unit Testing” would be a good start. It can be easily embraced without significant impact on the process. “Automated Unit Testing” is a process where programmers write test cases for their code using testing frameworks such as the XUnit framework. Ideally, the test cases are written first followed by the implementation of the production code. When all the tests pass, the implementation is complete.


Challenges in “Automated Unit Testing”
More than the impact on the process, the challenge is in getting teams to adapt to this process. In waterfall process, development teams typically provide the implementation and throw it over the wall for QA to test. Asking programmers to prove that the code works before handing it over to QA is a significant paradigm shift. Writing tests is also hard – especially if the code has not been written in a testable way. Developers may have to spend as much time writing the test cases as the production code itself. To most developers, this seems like a waste. However, this pays off in terms of producing code with fewer defects.


Better late than never
It’s never too late to write tests. Tests are most effective if they are written at the time the production code is being written. However, you could introduce them even as defects are being fixed for existing code. When a defect is fixed, insist that tests be written for it. This allows for an incremental creation of test suites. Test suites allow for your code to be flexible. Their presence lets you re-factor with confidence.


Conclusion
Automated Unit Tests are a pillar of agile teams. For teams that want to adopt agile practices, this is a great way of adopting one of its pillars without disrupting existing processes. It’s never too late to adopt Automated Unit Testing. Embracing it in Waterfall also sets the stage for adopting the other important aspects of the Agile methodologies.

Embrace Upstream Testing in Waterfall

Upstream testing is a process where developers and testers work together from very early in the development cycle - even as features are being built. This is a flavor of an agile development methodology that can be embraced even in a waterfall process.


Introduction
Agile software development teams have reaped significant benefits by adopting the methodology over the waterfall process. However, there are significant challenges in moving from waterfall to agile. Most teams that make the change adopt a home grown hybrid model eventually leading to a complete agile methodology. If you are in a team that is practicing waterfall and you want to be more agile, consider adopting “Upstream Testing” in your current process.

How testing is usually done in waterfall
In ideal waterfall, features are delivered to QA after they are completely built - in one fell swoop. More commonly, features are delivered in interim builds to QA for testing. QA tests the interim build and reports defects on them. These defects are triaged, prioritized and assigned to developers for fixing in future builds. This is usually a slow and expensive process. Often times, the developers have moved on to other features building, sometimes, on the existing defects. Fixing the defect now is more expensive and risky. Additionally, in many situations QA is not able to complete the testing of features developed in the build. As additional builds are delivered, QA is often behind in testing. The later a defect is fixed, the higher the cost. Unfortunately, since the risk of fixing is also higher with the passage of time, many defects are deferred to a later release – which is effectively a “death sentence” for the defect. The reality is that it will never get fixed, resulting in an inferior product to the customer.

Embrace Upstream Testing
What can waterfall organizations to do in the short term to improve the quality of their deliverables? Adopt “Upstream Testing” in your process. “Upstream Testing” is a flavor of agile testing methodology that can be embraced without causing major disruption in existing processes. Very often it can be done without even the involvement of senior management.

In this process, QA works with software development even as features are being iteratively developed. As the developers are checking in code, the QA resources are testing the functionality on daily builds. The daily builds are not formal builds that are handed off to QA with all the bells and whistles. Instead, it is a build that is used by development to test the validity of their integration and check-ins. QA and development work off the same build. The real differentiator in this process is what happens when QA finds an issue. The tester simply reports this to the developer without registering the defect in the corporate defect tracking tool. The developer and tester agree on a fix and it is implemented right away. The tester tests the fix and the developer moves on to other features confident that not only has he unit tested it but that it has even been approved by a tester. When the formal build is done and handed off to QA for intensive and regression testing across many platforms, there are no surprises to QA and quality is already very high.

Note that there aren’t any significant deviations from how development and testing is done. The QA and development teams continue to report to their respective managers. The organization hierarchy is preserved while still injecting agility into the process. Apart from the development and QA teams, hardly any other organization is functionally affected – in contrast to a full agile methodology which has a ripple effect throughout the organization.

Benefits of Upstream Testing
Since the tester and developer are working together, the quality of the deliverable is being monitored continuously – almost on a daily basis. Defects are found and fixed early. It is extremely cost effective to the organization to have this done without going through defect reporting, triages, defect assignment, Change Control Board’s and such ceremonies. Defects that might get deferred are actually fixed – because it is possible to do so. The result is a higher quality product, delivered earlier in a more cost effective manner. The social engineering aspects of “Upstream Testing” cannot be underestimated. The tester gets to understand the complexity of development and the developer begins to appreciate the value of the tester in making the developer look good.
Development and QA start beginning to look like a team working towards a common goal.

Operational Challenges in Upstream Testing
Many organizations are set up functionally. Thus, the QA and development teams report to different directors who may ultimately report to the VP of Engineering. The very nature of this organization sets up different priorities for QA and development. Very often, one of the metrics used to measure the performance of QA teams is the number of defects filed per release of a product. It is not unusual to see colorful graphs of defects found outside the QA director’s office while the development director’s office proudly displays the defects fixed.

In “Upstream Testing” the defects filed are far fewer – because they are caught and fixed informally. The management of QA and development need to arrive at an understanding of a more mature and relevant measure of performance. This can often be challenging and needs good understanding and common goals between development and QA managers.

Conclusion
Upstream Testing is an agile practice that can be fairly easily introduced to teams following waterfall. It is not disruptive and fits in well with waterfall processes. It allows for defects to be caught earlier resulting in cost efficiencies. Apart from higher quality deliverables, the effect of the social engineering it introduces between development and test teams is high. If your team is following a waterfall process and looking to adopt more agile processes, consider “Upstream Testing”. It is an enabler for moving towards full fledged agile methodology in the future, while solving real problems today.

Embrace the daily stand-up in Waterfall

The Daily Stand-up meeting is the heartbeat of Scrum, an Agile methodology. This is a process that can be easily embraced by teams following the waterfall process.

Introduction
Agile software development teams have reaped significant benefits by adopting the Scrum methodology. However, there are significant challenges in moving from a Waterfall process to an Agile process. Most teams that make the change adopt a home grown Hybrid model eventually leading to a complete Agile methodology. It has been argued by many, that a swift change from one process to another is better than a gradual change. While there are merits to the argument, the reality is that most organizations and people take time to embrace change. It is better to take the time to win hearts and minds than to force adoption of a new methodology. Change that is willingly embraced is change that can be sustained.

Let’s say you want to adopt some Agile methodologies in your team that is currently following the waterfall process. What aspect of it should you adopt? My suggestion is to introduce the “Daily Stand-up” in your team. The “Daily Stand-up” is the heartbeat of the Scrum Agile methodology. See here for an explanation of this process:

http://www.mountaingoatsoftware.com/daily-scrum

Ask the team if they are willing to try this out for a couple of weeks. Also choose a time that is most convenient to them. Most teams will be willing to try this out. You can (and I recommend that you should) even offer the following incentives to the team, if it is in your power:

Cancel the weekly status reports
Most programmers dislike writing status reports. They’d rather write code. With the daily stand-up you probably don’t need weekly reports anymore. You are getting the status daily! You can use the information at the daily stand-up to update the project plan and produce a burndown chart to track progress.

See here for description of a burndown chart:
http://www.mountaingoatsoftware.com/release-burndown

Your team members will love you for canceling the weekly status report!

Cancel the status meeting
Most waterfall teams have weekly status meetings that last for an hour or more. Most programmers dislike long meetings. Since you are getting a daily update, you probably don’t need this meeting anymore. Cancel it.

Your team members will love you for canceling the weekly status meeting!

What’s the gain from the daily Stand-up?
As a Project Manager you have a daily update on the progress. This is better than the weekly update from the original status meetings. You have information early that you can act upon. There is less formality in a stand-up – programmers tend to like that. Information exchanged in a stand-up often results in offline warm & fuzzy communication.

Conclusion
The daily stand-up contributes heavily to the success of an Agile methodology such as Scrum. If the 80/20 rule were to be applied to Scrum, the daily stand-up would show up on the “20” side. It can be easily incorporated into the Waterfall process with all the resulting benefits. Embracing it in Waterfall also sets the stage for adopting the other important aspects of the Scrum Agile methodology.