Study Notes: Google Machine Learning Engineer Certification

Section 1: ML Problem Framing

1.1 Translate business challenge into ML use case

Defining business problems

Identifying nonML solutions

A good rule of thumb is to have at least thousands of examples for basic linear models, and hundreds of thousands for neural networks. If you have less data, consider a non-ML solution first.

ML is better at making decisions than giving you insights. If you have a bunch of data and want to find out “interesting” things about it, statistical approaches make more sense.

Never launch a fancy ML model that can’t beat a heuristic. The exercise of making a heuristic often will help you identify good signals in your ML model.

Defining output use

Consider how you will use the predicted outcome in your product.

• Will it be presented immediately to the user in a UI?
• Will it be consumed by subsequent business logic?
• What latency requirements do you have?

Managing incorrect results

Identifying data sources

Provide answers to the following questions about your labels:

• How much labeled data do you have?
• What is the source of your label?
• Is your label closely connected to the decision you will be making?

1.2 Define ML problem

Defining problem type (classification, regression, clustering, etc.)

Defining outcome of model predictions

Defining the input (features) and predicted output format

1.3 Define business success criteria

Success metrics

Quantifiable: Your success and failure metrics should be phrased independently of your evaluation metrics such as precision, recall or AUC. Instead, specify the anticipated outcomes.

Measurable: A measurable metric provides enough information for successful real-world evaluation. For example, a system monitoring the health of an orchard might want to reduce the fraction of sick trees that die. But if you can’t measure how many trees are sick, this is not a useful metric. Ask the following questions:

• How will you measure your metrics?
• When can you measure your metrics?
• How long will it take to know whether your new ML system is a success or failure?

Ideally, you want to fail fast. Watch for too little signal in your data, or data that isn’t predictive, to determine if your hypothesis might be wrong. Failing fast will enable you to revise your hypothesis earlier in the process and prevent lost time.

Key results

Determination of when a model is deemed unsuccessful

Failure Metrics: The failure metrics are important, that is, how will you know whether the ML system has failed? Failure may not only be caused simply by non achievement of a success metric.

1.4 Identify risks to feasibility and implementation of ML solution

Assessing and communicating business impact

Assessing ML solution readiness

Assessing data readiness

Aligning with Google AI principles and practices (e.g. different biases)

Types of Bias

• Reporting Bias: occurs when the frequency of events, properties, and/or outcomes captured in a data set does not accurately reflect their real-world frequency. This bias can arise because people tend to focus on documenting circumstances that are unusual or especially memorable, assuming that the ordinary can “go without saying.”
• Automation Bias : a tendency to favor results generated by automated systems over those generated by non-automated systems, irrespective of the error rates of each.
• Selection Bias （Coverage bias: Data is not selected in a representative fashion. Non-response/participation bias: Data ends up being unrepresentative due to participation gaps in the data-collection process. Sampling bias: Proper randomization is not used during data collection.)
• Group Attribution Bias (In-group bias: A preference for members of a group to which you also belong, or for characteristics that you also share. Out-group homogeneity bias: tendency to stereotype individual members of a group to which you do not belong, or to see their characteristics as more uniform.)
• Implicit Bias (confirmation bias, experimenter’s bias):occurs when assumptions are made based on one’s own mental models and personal experiences that do not necessarily apply more generally.

Identifying Bias

• Missing Feature Values
• Unexpected Feature Values
• Data Skew

Evaluating for Bias

When evaluating a model, metrics calculated against an entire test or validation set don’t always give an accurate picture of how fair the model is.

You can use confusion matrix to check how the model performs on each subgroups.