https://www.kaggle.com/julianmukaj - https://www.linkedin.com/in/julian-mukaj-31766487/

Thank you to CrunchDAO and ADIA Lab for organising this competition and to all involved for their efforts in running operations and support of it.

Introduction and Problem Statement

https://hub.crunchdao.com/competitions/structural-break

Structural breaks are a critical concept in time series analysis and econometrics. They refer to sudden changes in the underlying structure of a time series, which can significantly impact forecasting and modelling. This global data science competition offers a $100,000 prize pool and the chance to solve a real-world problem: detecting structural breaks in time series data - a challenge with applications in finance, climate, healthcare, and beyond.

The task of this competition is structural break verification: the goal is to determine whether a structural break has or hasn’t occurred at a specified point in each time series you will be given. Our detection algorithm has to be designed to take as input a time series and output the likelihood (a score between 0 and 1) of a structural break at a given change point period.

The Training Data has the following structure:

• A MultiIndex with two levels:
  • id: Identifies the unique time series (each time series has a unique ID)
  • time: The timestep within each time series
• The columns include:
  • value: The actual time series value at that timestep;
  • period: A binary indicator where 0 represents the period before the boundary point, and 1 represents the period after the boundary point. The structural break occurs at the change in value, but it may take some time (values) to become detectable/apparent.

Solution Quick Overview

The final solution submitted is a bagged 10 KFold ensemble model of 3 GBDTs (LightGBM, XGBoost and CatBoost) consisting of 231 features that were selected from a larger pool of ~1000.

The model was first trained on all features. The feature importance was tracked per fold per model and then the importance scores were averaged for all the features per model, the top 100 was selected from each model and the union of all 3 different models made the final feature selection from the KFolds. The variance features based on the various transformed series were added onto these after because they reduced the Stdev of the KFold scores. Optuna was used to hyper parameter tune each model to maximize the overall AUC score from the 10 KFold. The final prediction is the mean of all models within the bagged ensemble model.

🚀 Final Solution CV: 89.47% (Min 87%, Max 91.5%, Stdev 1.39%) AUC

🌌 Public Leaderboard (6th): 88.38% AUC

🏅Private Leaderboard (4th) : 89.32% AUC - Prize Winner

Top Features Breakdown

This is a breakdown of the category groups of features that were selected post feature selection process - beyond a subset of the top ~150 features, most features after provided very minor Overall AUC boosts in the Cross Validation but reduced the standard deviation of the AUC scores between folds and brought up the minimum AUC too so expectation is they do add robustness even if not improving the final CV score.