휴일 예측 정확도를 높이기 위한 Cascade ML Approach

왜 이 글을 썼는가?

• At DoorDash, we generate supply and demand forecasts to proactively plan operations such as acquiring the right number of Dashers (delivery drivers) and adding extra pay when we anticipate low supply. It is challenging to generate accurate forecasts during holidays because certain machine learning techniques (e.g., XGBoost, Gradient Boosting, Random Forest) have difficulty handling high variation with limited data
  

Tree Model의 한계성

• In particular, tree-based models fail to capture high variation during holidays even with one-hot encoding because each holiday only appears once a year. Please see a simplified example of this in Figure 2 below.
• 휴일별 특성을 담는 것이 아니라, 휴일들이 특정 기준에 의해 Terminal Note에 묶여져 평균으로 묶일 수 있게 된다.  7월 4일, 추수감사절, 크리스마스 감소추세가 동일하게 계산된다.

Approach

• reframing, rebalancing, and multilabel ensembles 등이 그간 쓰였음
• 일반적인 방식은 Cascading ML Approach
• ML문제를 나눠서 해결한다.
• Machine Learning Design Patterns
• Holiday, Regular Day를 나눠서 예측 →  이 경우 Time Series에서는 사용하기 어려움
• To prevent losing the most recent information, we implemented an augmented version of a cascade design pattern tailored for time-series forecasting. As shown in Figure 3, instead of building two models, we correct the actual series by removing holiday impact so that recent historical information is available when producing forecasts following a holiday.

As-Is

• GBM with wMAPE
• 크리스마스 언저리에서 하락하는게 보임
  

To-Be

Calculating holiday multipliers:

• we leverage parallel computation in Spark and store the holiday multipliers into a table for the GBM model to use in the preprocessing step.

Preprocessing holiday multipliers and model training

• we train and store the GBM model.

Generating forecasts and post-processing

• Because the GBM model was trained with non-holiday data, our initial forecasts do not take holidays into account.
• For example, if the non-holiday forecast for Thanksgiving Day is 150 and the associated multiplier is 1.5 for a given starting point, the final forecast is 100 (=150/1.5).

- 운영및 재무팀 개입을 최소하기 위한 구조에 적합한 형태(확장 가능하기 용이)
- 성능 개선을 이끄는데 성공

한계

• Switchback Experiment Design 채용 불가
• 네트워크효과로 인한 간섭가능성 때문에
• here is more info on switchbacks
• 목표 설정의 어려움(비즈니스가 더 우선순위이니...)
• Consequently, we had to reframe our experiment’s goal. Rather than accurately measuring the cascade approach’s impact, we decided to focus on ensuring that it does not worsen quality. If we could keep metrics flat without seeing quality degrade across a set of selected holidays, we could then make the decision to ship.

결과

• Even though calculating the holiday multipliers seems like an extra step, they helped us to explain the forecasts to our stakeholders, which built more trust in our models and accelerated the transition to end-to-end ML solutions with minimal manual intervention.
• ML practitioners should evaluate the pros and cons before deciding to use the cascade design pattern.  If the process adds significant complexity to the model architecture with only limited gains in target loss, it may not be worth the effort.

References

• How DoorDash Improves Holiday Predictions via Cascade ML Approach