Agile vs Waterfall Project Management Methodologies

This is the second in a series of articles from City Integration and Limina written with the intention of demystifying some of the complexities around Investment Management platform and system implementations.

In our previous article, we outlined some thoughts around the implementation approach and discussed the merits of a Big Bang vs Phased approach. In this article, we take that thinking further and focus on implementation methodology.  Another key decision to be made at the outset of any IMS implementation.

Implementation Approach vs Implementation Methodology

It is often the case that the implementation approach and implementation methodology are confused and conflated. Big Bang does not automatically mean Waterfall, and, equally, Phased does not mean Agile, so it might be helpful just to take a moment to clarify the distinction between the two terms:

Implementation Approach: This is how both the project teams and the outside world see your project. The main slide at any Steering Committee or external communication will show the key timelines and deliverables, and it will be clear to all how you intend to deliver the change, either in one hit or incrementally

Implementation Methodology: This is how you plan to run and deliver the work that makes up the project. This will be less visible to people not directly involved in day to day delivery of the project and is more about how you organise and schedule the work.

The main methodologies used in financial services tend to be "Waterfall" or "Agile". Other approaches such as Six Sigma and Kanban are available but these tend to be more specialised and less utilised by asset managers and investment managers, so for the purposes of this article we will focus on Waterfall and Agile only.

What is Waterfall Project Management?

Waterfall project management is a sequential and structured approach to software development and project execution. It follows a linear progression, where each phase of the project is completed before moving on to the next. The methodology typically includes distinct phases such as requirements gathering, design, development, testing, and deployment.

The Waterfall approach can be broken down as follows:

This methodology was largely defined by the constraints of the mainframe platforms prevalent at the time.  These platforms were typically only subject to major upgrades or enhancements every 1-2 years or more so this structured approach fitted the release schedules

As with any approach, there are always Pros and Cons associated with Waterfall project management:

Pros of Waterfall Methodology

1. Clear and structured: The waterfall methodology follows a well-defined and structured approach. It allows for clear project planning, requirements gathering, and documentation, making it easier to understand and manage the project.
2. Easy to manage: The sequential nature of the waterfall methodology makes it easier to manage and control the project. Each phase has specific deliverables and milestones, allowing for better project tracking and progress measurement.
3. Documentation centric: Waterfall methodology emphasises comprehensive documentation at each stage. This can be beneficial for future reference, maintenance, and knowledge transfer, ensuring that the project is well-documented from start to finish.
4. Suitable for stable requirements: It is most effective when the requirements are well-defined and stable up front. If the project scope is clear and unlikely to change significantly, the waterfall methodology can be a good fit.

Cons of Waterfall Methodology

1. Lack of flexibility: The waterfall methodology is inflexible once a phase is completed. It does not easily accommodate changes or adapt to evolving requirements. If changes are needed, it can be time-consuming and costly to go back to a previous phase.
2. Limited customer involvement: Customer involvement is typically limited to the early stages of requirements gathering. This can result in a lack of customer feedback until the later stages of the project, potentially leading to misalignment between the final product and customer expectations.
3. Risk of late issue identification: Since testing and validation occur towards the end of the waterfall process, there is a risk of late identification of issues or defects. This can lead to higher costs and delays in resolving problems.
4. Longer delivery cycles: Due to the sequential nature of the waterfall methodology, the entire project is completed before delivery. This can result in longer delivery cycles, especially for large and complex projects, as the final product is not available until the end of the project.
5. Limited scope for learning and iteration: The waterfall methodology does not provide much room for learning from mistakes or incorporating new insights during the development process. It may not be suitable for projects that require a more iterative and collaborative approach.

What is Agile Project Management?

Agile project management is an iterative and flexible approach to project execution. It emphasises collaboration, adaptability, and customer involvement throughout the project lifecycle. Instead of following a linear progression, Agile breaks down the project into smaller iterations called sprints, with each iteration delivering a working increment of the product. It encourages continuous feedback, allows for changes and refinements to requirements, and promotes cross-functional teams.

The Agile approach can be broken down as follows:

As with the Waterfall approach, Agile project management comes with its own set of pros and cons:

Pros of Agile Methodology

1. Flexibility and Adaptability: Agile methodology allows for flexibility and adaptability to changes in requirements and project scope. It enables teams to respond quickly to customer feedback, market trends, and emerging needs, resulting in a higher likelihood of delivering a product that meets customer expectations.
2. Customer Collaboration: Agile emphasises regular customer involvement and collaboration throughout the project. This helps in better understanding customer needs, gathering feedback, and ensuring that the final product aligns with their expectations. It also increases customer satisfaction and reduces the risk of delivering a product that doesn't meet their requirements.
3. Iterative Approach: This breaks down the project into smaller increments or sprints (usually between 2-4 weeks per sprint). This allows for faster delivery of working product increments, enabling early and continuous feedback, reducing the time to market, and increasing stakeholder engagement.
4. Transparency and Visibility: Agile promotes transparency by emphasising open communication, regular status updates, and visible progress tracking, especially when combined with integrated mass market development tools and trackers. This allows stakeholders to have a clear understanding of project status, progress, and any potential issues, facilitating better decision-making and risk management.

Cons of Agile Methodology

1. Project Scope and Requirement Changes: While adaptability is a strength of Agile, frequent changes in project scope and requirements can be challenging to manage. It requires effective prioritisation, communication, and coordination to ensure that changes are implemented efficiently without negatively impacting the project timeline or budget.
2. Team Experience and Expertise: Agile projects heavily rely on the skills, experience, and collaboration of team members. If the team lacks expertise or experience in Agile practices, it can lead to difficulties in adopting and implementing the methodology effectively, potentially impacting project outcomes.
3. Resource Allocation and Planning: Agile requires ongoing resource allocation and planning for each iteration or sprint. This can be challenging when dealing with limited resources or when multiple projects or priorities need to be managed simultaneously. Careful resource management and coordination are essential for successful Agile implementation.
4. Documentation and Formality: Less emphasis is placed on comprehensive documentation in an Agile deployment when compared to traditional methodologies. While this can be an advantage in terms of flexibility, it can also lead to a lack of documentation for future reference or knowledge transfer. Proper documentation practices need to be in place to ensure that critical project information is captured and preserved.
5. Fragments the bigger picture: Agile is focused on the immediate deliverables as opposed to a strategic vision. As a result, it may sometimes overlook the long-term strategic goals of the project or organisation. Agile methodologies prioritise delivering incremental value in shorter time frames, which can be advantageous for responding to changing market demands. However, it is essential to strike a balance between short-term deliverables and the overarching strategic vision to ensure that the project aligns with the broader objectives and maintains a sustainable trajectory for long-term success.

Waterfall vs Agile Methodology Which is Better?

There is rarely a simple answer - a straightforward technical upgrade with little or no business impact (e.g. a new website, or change of data provider) may well suit a purely Agile approach, whereas a small-scale project where requirements are clear from the outset and unlikely to change may suit a Waterfall style delivery.

Over the past 10-12 years with improvements in platforms, cloud-based hosting, SaaS etc, as well as major advances in development and project management tools such as Jira, Asana, Trello etc. makes the Agile approach more popular. The Waterfall methodology is now considered to be somewhat clunky and outdated.

Realistically, and from what we see in practice, the complexity of financial services organisations tends to see them utilising a hybrid approach for medium to large projects, applying elements of both methodologies with the precise balance being dependent on the specifics of the project and the impacts across the business, operations and technology.

Based on our expertise and irrespective of the methodology used, several vital factors consistently play a central role in every project and must be considered when determining the approach:

• Requirements Definition: It is essential to establish the overarching objective(s) of the project at the initial stages of the process.
• Planning: Breaking down the requirements into manageable phases allows for effective execution and progress tracking.
• Disciplined Teams: The success of the project relies on efficient program and project management, involving analysts, project managers, developers, and testers.
• Engaging Key Stakeholders: Managing the interaction with important stakeholders and sponsors is crucial, as it ensures their support and commitment to the project.

Successfully navigating these options is what will help define the approach and is where City Integration can help. Our specialist professionals have 25+ years of experience delivering complex programmes of work at blue-chip organisations such as investment banks, investment managers, central banks and sovereign wealth funds, supported by our fully-qualified team of analysts, developers, scrum masters and project managers.