Kanban for AI Projects
How Kanban Enhances Efficiency and Flexibility
My journey with AI projects started almost 4 years ago, after more than 10+ years of managing various IT projects I joined the AI team as a project manger. The initial experience related to AI projects was markedly challenging; the strategies and techniques methodologies I had previously relied on for IT projects fell short.
I quickly learned that managing AI projects is fundamentally different from traditional software development. AI projects demand unique approaches and skill sets due to their dynamic and experimental nature.
As more organizations look forward to implementing AI for various purposes, including process automation, productivity enhancement, and cost and time reduction, effective management of AI projects becomes essential.
The success of managing such projects largely depends on the capability of those leading and executing these projects. Thus, it is important to understand the nuances of managing AI and Data Science projects and leverage the skills and techniques that are the most suited for the specific needs.
Unique Challenges in AI Projects
The majority of AI projects are marked by a high rate of unpredictability. From ‘Day 1’, it’s uncertain whether the AI solution will be developed, delivered, or how long it will take. And it is related to data that plays a crucial role in AI projects. The quality and quantity of data are pivotal, significantly impacting the project’s success or failure.
AI projects often resemble research and development (R&D) activities, requiring extensive experimentation. Predicting outcomes and estimating task duration is notably challenging. Many tasks termed ‘research tasks’ require a review of the existing literature to check if a similar problem has been addressed and, if so, how.
AI projects are dynamic and can conclude prematurely. For example, one of my first projects ended at the early stage due to limited datasets and poor data quality making it impossible to predict consumer goods production outcomes. The decision was made to put the project on hold. The recommendation for the customer was to accumulate sufficient data of the required quality and quantity before restarting it again.
Additionally, AI projects are innovative projects. They have many elements in common, like models, frameworks, and the customer needs. However, the variety and scope of datasets mainly set them apart from one another.
Leveraging Kanban for AI Success: Real-World Examples
The Kanban Method stood out as a way to manage knowledge work. Also, it has successfully stood the test of time providing value across various business domains. What is important, the Kanban method is highly adaptable and can be effectively implemented in AI projects, bringing numerous benefits. I will highlight a few examples below.
1. Kanban for AI Teams Looking for More Freedom
At the start of every project, including AI projects, teams form and learn how to collaborate and also discover their knowledge level in applying Agile methodologies, which can vary. Thus, Kanban can be particularly beneficial for such teams, offering a less formal and complex approach thanks to the “start where you are” approach.
In one of my projects for video object detection, we initially followed strict Scrum guidelines, which turned out to be counterproductive. The intensive requirements and rigid ceremonies slowed the team down. The team’s motivation also decreased. Consequently, we decided to shift to Kanban early in the project, granting the team more freedom and simplifying the process. This proved to be a very good step that led to improved team motivation, better cooperation, and work delivery progress.
2. Kanban for the data science (DS) projects
Data Science projects are part of a larger AI project ecosystem. They lay the groundwork for AI systems through data analysis, statistics, and visualization. This foundational work is crucial for understanding and preparing data before its application in more sophisticated AI technologies, such as machine learning and deep learning. Typically, data science projects serve as the initial phase in the development of an AI solution.
Estimating the time required for data science tasks can be challenging, given the variability in data complexity, project scope, and specific objectives. Kanban offers a more flexible approach, providing data science teams with the leeway to carry out their tasks without the constant pressure of meeting sprint deadlines. This flexibility makes Kanban an excellent approach for Data Science projects. Research studies indicate Kanban ranks as the third most effective approach for these projects, following CRISP-DM and Scrum.
In one of the data science projects I led, the goal was to extract insights from various data sources to determine whether operation optimization in a warehouse could achieve a 30% improvement with the use of AI technology. This venture into a new domain meant that the team needed time to explore without the burden of task estimation and deadline pressures. During the initial phase, we adopted Kanban. This approach proved to be beneficial as it allowed for greater flexibility in delivering results, given that the client did not require regular verification.
3. Kanban for building AI continuous improvements
Kanban stands out as a beneficial system for enhancing collaboration and coordinating tasks outside the direct AI system development. Its fundamental principles – visualizing the workflow and minimizing work in progress (WiP) – were pivotal in my team’s successful management of a series of tasks associated with developing AI consulting services. The motivation for adopting Kanban was the ongoing status of numerous tasks remaining in progress. Facing this challenge, we turned to Kanban that helped us to streamline our workflow and alleviate bottlenecks, effectively reducing WiP and facilitating uninterrupted progress.
While the team was geographically dispersed, implementing a virtual Kanban board was an essential requirement. This virtual setup ensured that every team member remained informed and engaged, regardless of their location. Daily meetings became a cornerstone of our task management strategy, enabling continuous communication, prompt issue resolution, and consistent momentum. This approach not only optimized our workflow but also fostered a culture of transparency and collective accountability.
Key Takeaways: Insights on Using Kanban in AI Projects
Given the dynamic and experimental nature of AI/DS projects, utilizing Kanban or a hybrid approach with other elements (e.g. Scrum) can offer effective alternatives for managing AI-related projects. The simplicity of Kanban and its strong alignment with Agile practices can be a viable solution for the complex and unpredictable nature of AI projects.
There is no one-size-fits-all framework in project management, especially in fields as rapidly evolving as Artificial Intelligence and Data Science. Kanban is not a replacement for a current framework or process. Rather it works with any existing process or framework and takes an evolutionary approach to improving what is already in place. Therefore, I believe that just as experimentation is crucial in AI projects, it’s equally important to experiment with management approaches for such projects. This strategy ensures that the AI project is managed effectively while remaining focused on delivering its primary objectives.
Get Started with Kanban
Ready to enhance efficiency and flexibility in your AI projects? Dive into the world of Kanban with our comprehensive course, Introduction to Kanban. Learn how to streamline workflows, minimize bottlenecks, and foster collaboration within your team. Take the first step towards mastering Kanban and revolutionize your approach to AI project management today!
About the Author – Ewa Szulik-Stach
Ewa is an AI Manager with over 15 years of expertise in project management, specializing in AI projects and initiatives. An AI enthusiast at heart, she excels in employing Agile, Scrum, and Kanban methodologies to boost team productivity and achieve project outcomes. Ewa holds various certifications, blending her knowledge of AI with her expertise in project management. She has a deep passion for the future of AI, with specific interests in AI education and its application for children.
