Phase 4: Model Development in CPMAI
In CPMAI methodology, phase 4 is where teams select algorithms, train models using their data, and finally, adjust the hyperparameters. During this phase, the project manager, as the scope, resource, and business-alignment manager, oversees the build. This is where many people would define an AI project; however, in CPMAI, phase 4 is just one of six, and it is not the most important phase of the project, considering a good chunk of the model’s quality and performance depends on the dataset preparation from Phase 2 and Phase 3.
Why is Phase 4 important?
The objective of the Model Development phase is to design a model using a prepared and cleaned dataset that has the potential to satisfy the success criteria defined in the Business Understanding phase. This phase essentially aims to create a model as described in the business requirement, and are aligned with the original business goals. This has two very important implications. First, if a data science team is left to their own devices, they may lose focus of what the model is designed to accomplish in the business domain. As a result, the model would possess no meaningful value in the context of the business.
What are the components of Model Development, and what are their steps? 
There are 5 main steps in building a model. These are Algorithm selection, model training, hyperparameter tuning, tracking experiment results, and finally building and executing the model. Each step brings you one step closer to building a model that can be evaluated.
Building a model requires task management, experiment tracking, scope management, resource management, and scheduling because it is a time consuming and cost intensive process.
Selecting an Algorithm
You should justify why you have selected an algorithm based on the given business problem and the available data. A less complicated model, which is interpretable, is preferably correct and requires justification over a more complicated model that may be more accurate, but outperforms the less complicated model and is therefore, meaningless.
You need to determine if the problem is classification or regression, etc. based on the findings you made during business understanding.
Consider the amount of data available versus the algorithm’s training dataset (some algorithms need a large amount of data to be reliable). Also, think about interpretability, especially for algorithms that you will have to explain to the stakeholders/auditors, and for which the rationale behind the model’s decisions will need to be justified.
Justify Algorithm Selection. Note that your choice may have to be defended again during Model Evaluation if performance gaps are reported.
How do you document and explain progress in model experiments?
How do you manage to continually prioritize business goals over hyper parameter tuning?
What Strategies Do Project Managers Employ During Model Development?
What Outputs are Associated with Model Development? 
The primary output of Model Development is a trained model prototype, alongside training results and a justification for the final model configuration selected for evaluation. This output is then scored against an unlabelled dataset and is validated for bias before it is eligible for deployment.
|
Deliverable |
Contains |
|
Trained Model Prototype |
Final model configuration, experiment history, and rationale |
What happens When Model Development Loses Alignment With The Business Objective?
One of the failure patterns of CPMAI is when model development drifts from the business objective. During these scenarios, the model progresses to the Model Evaluation stage, but is tuned for another metric which is not related to the objective.. Consequently, the model development work has to be initiated again from either the business understanding or data preparation phases, which are two of the most time-consuming phases of the CPMAI framework. For this reason, the project management framework of CPMAI gives a lot of emphasis to the project management role, to monitor the development of the model to ensure that model development work is not unsupervised and is undertaken by the technical team.
The model achieves a high score in the Model Evaluation stage; however, the model does not impact the metric for which it was built to articulate.
Unmanaged scope encourages the technical team to address issues which are not related to the business goal.
Long cycles of iterations on the model lead to exhaustion of budget for the phase of Model Operationalization.
How is Model Development Related to the PMI-CPMAI Exam?
Model Development relates to Domain IV of the PMI-CPMAI exam: Manage AI Model Development and Evaluation. This domain encompasses about 16% of the exam questions. According to the PMI Exam Content Outline, the oversight of model development and evaluation falls within this domain, so candidates may have to answer questions that will evaluate their experience and judgment with regard to selecting algorithms and scoping and the usage of project resources for the go/no-go decision.
Where is Model Development located Within the Full CPMAI Methodology? 
Model Development is the fourth of the six phases in the CPMAI methodology. For further details regarding the other phases of the CPMAI methodology, refer to The CPMAI Methodology: 6 Phases Explained.
Where would one learn to apply Model Development Oversight in a Practical Context?
PMTI provides live and interactive classes to teach project managers to oversee the development of models and to train them without the need to write the program code.
Phase 4: Model Development FAQs
Do project managers need to have coding skills to perform the Model Development phase?
No. When developing models, the CPMAI framework offers a project management viewpoint and does not encompass the requirement for programming.
What is experiment tracking and why is it critical when developing a model?
Experiment tracking means documenting each iteration of model training using various configurations, data, and outcomes. If teams do not conduct experiment tracking, they cannot justify or reproduce how a certain model configuration was selected.
How can we know when which model iteration stagnates?
Set a threshold for diminishing returns on the success criteria defined in Business Understanding, instead of implementing continuous improvement.
What PMI-CPMAI exam domain captures skills pertaining to Model Development?
Domin IV, Manage AI Model Development and Evaluation which is about 16% of the exam.
