Agile is not Enough: the Case for using Waterfall Techniques in Scrum Projects
Status Quo: Transitioning from Waterfall-only to Agile Projects
In today’s IT world, we are seeing ever more Agile projects being started. Usually those new, mostly Scrum-based IT projects keep replacing the elder approach, which is called “Waterfall.” This is due to its staging of Requirements Engineering first, planning second, architecture, code development, testing, documentation, etc. — all of it happens in sequence. The idea in earlier days was, to have one stage completed and then it’s respective output “pours” into the next stage — like a series of waterfalls.
Those Waterfall projects had their sound proof and reasonable logic behind them. Those projects used to be rather heavy on their overhead, with a slightly larger team and hence required more planning beforehand. In result, they lasted longer and were more price-intense. The Agile approach however claims to have shorter planning phases beforehand, hence were more price-efficient and had a little leaner team. Additionally, in Scrum and Agile projects, the users (tend to) get feedback immediately.
In the worst-case in Waterfall projects, the feedback would’ve gotten in the very end — when everything is wrapped up and finished. As an analogy: the house is planned, constructed, painted and the movers just left. This is the most extreme form of feedback a user gets in Waterfall planning — too late for simple changes. So, any need for change takes us back to the drawing board. There is however an approach to mitigate this risk in Waterfall projects: it is called Continuous Delivery (CD). Here users get regular deployments of their prospective solution and test it in action.
What Exactly is Agile Short of?
As stated before, Agile projects have theoretical upsides being more efficient due to slightly smaller teams, due to smaller planning footprint and might deliver faster — at least compared to one monolith release or consolidated releases in a bunch right in the end of the software development life cycle (SDLC).
The traditional Waterfall approach has sound logic and reasonable assumptions behind it as well. Planning the Critical Path (CP) of all deliverables impacting the final delivery deadline, resource utilization, measuring, testing and keeping software quality high were important aspects in the past — and still are. This used to be the key differentiator between stellar and mediocre project managers. This used to be so crucial that no Computer Science student was able to graduate college or university without mastering those skills — at least on paper to a reasonable degree.
Therefore, I still see great value in those Waterfall techniques. The Agile methodology is sound as well. But I feel it is not enough. The reason for it is simple: I used to work in a Scrum project, which forfeited much of the Waterfall approaches without any replacement. To our Product Owner (PO), velocity and fast delivery was way more important than quality in our delivery. We got ourselves very heterogeneous development environments: the local development varied most from notebook to notebook — there have not been many constants between them. The test environment was then different from all development environments and differed from the Integration, Load test & Performance test environment, which was unlike the final Production environment. This is no best practice in any IT endeavor. Quiet the opposite holds true: consolidating systems is highly suggested. We ended up in unit and acceptance tests failing due to bad quality in our test environment setup. We needed days to figure out, the root problem was not in our code, data or its processing — it was the setup of the environment itself, which needed to be done in a hurry without proof of quality or following the software vendor’s advice. The result was, we searched for errors were there were none. Only the tests came out randomly due to the database cluster not being time-synced. This caused the results to be random-ish and hardly matching the expected outcomes. Fixing the test environment later, fixed the issues we faced with “random-ish” results.
Strategic planning also felt short in this project. We were planning everything in our four-week Sprints: everything. There was no more planning on top of that. It was only due to pure luck, three months before going into production, I learned we were not prepared going into production due to missing hardware. This order needed to be placed right now. So, we worked overtime to fulfill this request to get ourselves the hardware we needed for our production deployments. Going into product later was not an option due to a hefty million Dollar penalty and harsh public criticism we anticipated in the failure-scenario.
Our Product Owner wanted to keep up our productivity throughout any single developer — regardless of parallel roles any person might have had besides that. So, he or she kept us out of planning loops on purpose. We placed the order what we thought of being just on time through an extra-pricy urgency process. Later we figured out, my requirements were not ordered as-is: but “simplified.” Those simplifications resulted in an ambiguous order which saved the supplier carrying it out to save some money. But the setup was far from useable for us. The missing interaction with the supplier caused the issue to be discovered only after the request was fulfilled about six weeks before going to production.
When we discovered the infrastructure was no use for us, it took hardly any time to figure out the request was altered for the worse. So, I took out my original description for it, we re-ordered it correctly this time. Just days before going into production, our infrastructure was up and running — this time is was ready to go for us. There was no slack on time and quality in the end. We paid the infrastructure more than twice due to costly priority fees. On top of that, we paid with sleepless nights throughout fortnight’s time before we went into production. In the end, we were so lucky the supplier was just fast enough for us. I could totally see my architecture going down the drain for no good reason. So, this issue here was missing transparency and a lack of a communication channel.
Additionally, to save a little time, we only conducted one Retrospective rather late in the development phase. The need for changes was huge and the feedback harsh. As a result of that, the Product Owner insisted we never conduct another Retrospective in the future. Not a single thing changed on this project. The meetings were a mess before and kept being unorderly and ill-prepared. The requirements were too vague and project planning kept not being conducted from any person on the project.
On top of that, our Scrum project never made use of Agile methodologies like regular user feedback or self-improvement ceremonies. One of the costly things in any software project is to find and fix bugs — things working other than expected in the final product. The initial implementation in that case will be augmented by a Change Request (CR) which will result in a fixing implementation. Due to the fear of Change Requests, user feedback was hindered all the way through by our Product Owner and Scrum Master (SM). The latter tended to be a Project Manager more than a Scrum Master. From not setting up a test environment, not deploying to it and not feeding back the need for changes from testers in the end. Waiving feedback loops on purpose is not the Agile way — quite the opposite. So basically, we were doing a Waterfall project with Continuous Delivery (CD) rather than Scrum. An Agile development would have encouraged user feedback and possibly even higher quality through a sound Definition of Done (DoD) to keep the quality of deliverables high. A proper Definition of Ready (DoR) would’ve kept the User Stories (US) clear and concise for developers to work with them without much need to have additional feedback and clarifications.
This Spiel of trading delivery time, quality, scope and budget is best known from Waterfall projects. This is where Waterfall has some wiggle room — which is mostly hated by the business. On the other side, a fixed delivery deadline is rather not so Agile. With a must-deliver date, a Waterfall approach appears sound to go for.
An Important Aspect is Communication, Transparency and Regular Feedback Loops
Contrary to my experience in this software project, communication, transparency and self-improvement loops through feedback are vital parts of any Agile endeavor. We could have had an easier time meeting our objectives with some more degree of communication and transparency. But in the end, this problem could have occurred in Waterfall projects as well. In a Waterfall project however, the Project Manager is responsible for the success of the Engagement. In a Scrum or Agile project, the entire team fails. No single person is responsible for the Non-Functional Requirements. At least not by her/himself. In the end the team fails and the Product Owner stays empty handed. May the finger-pointing begin!
When there is an Architect for all Functional and Non-functional Requirements, this might most likely be her/his area of responsibility. Like in volleyball, there needs to be very clear responsibilities and areas of focus for any team member in software projects. Steve Jobs was a strong believer of the same and so am I. Whenever there is ambiguity, things go south. Either two or more people fight over competencies and the best solution or no one cares at all. Both are equally bad options. Like in volleyball, this point is wasted then when no single team player feels obliged to act — either by no one taking the ball or two people bumping into one another and potentially hurting themselves.
In other projects, I just came across Non-functional Requirements and architecture being waived entirely to keep the team’s velocity high. There was no stringency when it came to a long-term vision and nothing was constant. The infrastructure, production environment’s architecture and development tools were under constant change.
In another Agile project, the stakeholders had conflicting interests an no one consolidated them. As a result, we did the opposite of a Minimal Viable Product (MVP). Usually an Agile project is very good at delivering some quick wins fast. This works due to omitting non-critical features in the first iterations and only include must-haves initially. When neither scope nor the surrounding system’s architectures are fixed, the MVP is hardly feasible. Hence, we ended up with what is bad practice in Waterfall projects as well. We developed “golden handles.” Those features are nice to have — but their use is often doubtful. So, the development, documentation & test time is wasted right from the start. Features which might end up dead on arrival are usually an issue in Waterfall projects.
Our approach to Agile projects: Scrum & Waterfall Techniques Augment each other
Some colleagues of mine tend to not use either Agile or Scrum approaches. For best results they tend to make use of the best of both worlds. An Agile project, which is augmented using Waterfall techniques from the traditional software development life cycle (SDLC). Such a project might be a fully Scrum project with Daily Standup, Retrospectives, Sprint Planning, Sprint Review and Backlog Grooming sessions. All planning is under-taken in Agile project planning tools.
In combination, both methodologies’ blind spots are mitigated way better than with any approach by itself. Speaking in Data Science terms, we are performing Boosting here: take two or more good Classification or Regression models and come up with one more robust, improved prediction. This resulting forecast is likely to be (at least a little bit) better than any of the former models feeding into it by itself.
Additionally, we employ an Architect, Software Development Lead or Project Manager to have Non-Functional Requirements (NFRs) covered. The Functional Requirements (FRs) are covered in the Backlog and are maintained by the Product Owner, Proxy-PO or if in doubt by the Scrum Master (SM). This way we cover traditional project planning approaches from traditional SDLC and Agile methodologies at the same time. We feel rather glad having those processes in place. We can deliver high quality on budget and on time with high velocity. There might be an additional project plan with resource utilization plan, Critical Path, etc. But this is rather optional depending on the Project Manager or requirements from our respective client.
We feel very confident to deliver on time and on budget due to mostly the traditional SDLC approaches. This way, Waterfall techniques deliver the forecasts. The quality standards are defined by Agile methodologies like our Definition of Done, Definition of Ready and our test team. So operational planning in the near-future is solely subject to Agile approaches. We feel this combination really makes sense and helps a lot.
The Waterfall planning does not need to be perfect though. In fact, we apply changes to it ever so often. Hence the up-front plan is rather vague and rough. As we proceed on our endeavor, it keeps getting sharper and more robust. A lot from the initial, rough and uncertain plan has changed later in the project — but for the better.
Resumé
When applying Agile methodologies as-is, there is ample reason to miss something important down the road. As Scrum does focus primarily on Functional Requirements (FRs), the Non-functional Requirements (NFRs) might be missed out. Those NFRs were one major reason, important projects needed good Project Managers, Leaders resp. Architects. Additionally, the long-term planning is hardly subject to the Agile approach. In combination, both methodologies’ blind spots are mitigated way better than with any approach by itself. Speaking in Data Science terms, we are performing Boosting here: take two or more good Classification or Regression models and come up with one more robust, improved prediction. This resulting forecast is likely to be (at least a little bit) better than any of the former models feeding into it by itself.
