Description

Transformers have become a central architecture in deep learning, particularly for natural language and time series data. However, extracting causal relationships—such as identifying the influence of interventions or estimating treatment effects—remains a challenging task, especially in temporal contexts where confounders and non-stationarity may be present.

The objective of this thesis is to explore the integration of causal inference principles into Transformer models for time series data. The focus is on designing, implementing, and evaluating architectures that can learn causal temporal representations, possibly with extensions like counterfactual simulation or structural causal models.

Tasks

• Conduct a literature review on causal inference, attention mechanisms, and Transformer-based models for temporal data.

• Implement and evaluate selected models on benchmark time series datasets.

• Explore the integration of counterfactual or intervention-based reasoning in the Transformer framework.

• Analyze model interpretability and robustness to confounders.

• Develop your own ideas to improve existing approaches and investigate them.

Requirements

• Good understanding of the fundamentals of deep learning.

• Ideally: Experience with Python and one or more deep learning libraries (PyTorch, TensorFlow).

• Willingness to familiarize yourself with new subject areas.

Supervision

Shahenda Youssef

If you are interested, please send your CV to Shahenda.youssef@iosb.fraunhofer.de.