Unlocking financial insights: How Nubank powers personalized experiences with foundation models

Author: Austin McEver

At Nubank, we believe in fighting complexity, empowering people, and building customer-centered solutions. Our mission drives us to continuously innovate, especially in how we understand our customers’ unique financial journeys. Historically, financial institutions, including Nubank, have relied on traditional machine learning (ML) approaches, converting vast amounts of user data–like bank transactions, in-app events, and customer support logs–into manually engineered tabular features for predictive tasks such as risk prediction, fraud detection, and personalized recommendations [3, 4, 11]. While effective, this process is often labor-intensive and can struggle to capture the full richness and subtle nuances hidden within sequential data [3].

The challenge lies in moving beyond the limitations of static, aggregated features to build a dynamic, deep understanding of individual financial behavior. This is crucial because even slight improvements in core model performance can yield significant value [11, 4]. Inspired by breakthroughs in other domains like natural language processing (NLP) and computer vision, where models learn rich representations directly from raw data, we embarked on a journey to leverage Foundation Models for financial data [3, 4]. This is a key step towards Nubank’s AI-First vision [11].

Our journey accelerated significantly with the acquisition of Hyperplane in July 2024, integrating their expertise in developing large models for financial behavior with Nubank’s decade-long culture of productionizing technological innovation. Since then, platform and product teams collaborated to train and deploy large language models across several predictive modeling use cases, laying the groundwork for a more insightful and customer-centric financial future [11].

Since originally introducing nuFormer, we have deployed it to production to solve credit at scale in Brazil. We have also expanded the use of foundation models across Nubank, achieving impressive offline lifts in credit, lending, income prediction and X-Sell. This is an important step in showing our teams work can be delivered horizontally across Nubank

Architecting understanding: Parametrizing our foundation models

At the core of our approach is the adoption of transformer-based sequence models, which are adept at capturing long-range relationships in sequential data, a particularly useful trait for understanding the seasonal and cyclical variations in spending habits [1]. We’ve adapted these models to handle both textual and structured attributes from transaction data, an approach we denote as nuFormer [4].

Defining the transaction interface: from raw data to embeddings

Transformers operate on sequences of embeddings, so a fundamental step was defining how to convert a user’s transaction into a format the model can process [1]. Instead of assigning unique IDs to each transaction combination (which leads to massive ID spaces and cold start issues), we opted for a modified ”text-is-all-you-need” approach [1, 10].

We represent a transaction by its attributes, such as:

• Amount Sign: A token indicating positive or negative.
• Amount Bucket: Absolute amounts quantized into bins, each assigned a token.
• Month, Day, Weekday: Each represented with its own token.
• Description: Tokenized as natural language using a standard tokenizer (e.g., BPE) [1].

This modular representation allows for straightforward inclusion of additional categorical or numerical features and is more token-efficient than a fully text-based approach. We then concatenate these transaction strings for a member’s account, along with separator tokens, truncating to a predefined context length [1].

Scaling for insight: Pretraining and model parameters

Our pretraining involves standard language modeling tasks, such as next token prediction (NTP) or masked language modeling (MLM), where transaction tokens are embedded using a lookup table [1].

Pretraining Data Volume 

While even 5 million user + label rows show promising results, we’ve observed that scaling up to 20 million and 40 million rows yields incremental improvements in model performance. This indicates the models continue to benefit from broader exposure to transaction patterns. 

Context Length 

Understanding financial behavior often requires looking far back in a user’s history. Longer context lengths are generally better for capturing these long-range dependencies, but attention compute scales quadratically with context length, posing challenges such as slowed training and potential model divergence. Figure 1 illustrates the increased performance as context length increases.

Figure 1: nuFormer test AUC for different context lengths (512, 1024, 2048) with 24M and 330M models

Model Size 

We’ve demonstrated significant performance improvements by scaling model size. Initial benchmarks show distinct lifts moving from 24 million (24M) to 330 million (330M) parameters [1]. While 24M models are useful for fast experimentation and iteration, the 330M models generally perform better on our end tasks. Thus, we are actively working on scaling these models further, aiming for 700 million and 1.5 billion parameters, to unlock even more complex behavioral signals and emergent properties [3, 11]. Figure 2 illustrates the AUC performance gains from increasing model size.

Figure 2: nuFormer test AUC for different amounts of training data (5M, 20M, 40M, 100M) with 24M and 330M model

Crafting the user narrative: Strategic data selection

Nubank’s digital-first, multi-country approach generates vast amounts of rich transactional data, alongside other user interactions like app events [3, 11]. Constructing optimal “user narratives” for our foundation models involves treating data selection and representation as a hyperparameter search. The goal is to include as much useful information as possible, represented in as few tokens as possible, given the finite context window of transformers [6].

Diverse data sources and data source selection

We divide our data into various sources, and while we can pretrain on a superset, alignment between data sources used in pretraining and finetuning helps model performance. We employ a modular framework of “pre-processing modules” that convert transaction attributes into tokens [6]. For instance:

• Standard transaction attributes like amount, date, description, source, status, denial reason, and entry mode each have dedicated pre-processors.
• For unique financial products like Money Boxes (investment accounts), we include specific tokens for transaction_type, investment_type, and account_balance.
• Loan-related transactions leverage attributes like interest_rate, number_of_payments_made, and remaining_balance.

Crucially, we incorporate Credit Information System (SCR) data in our Brazilian models, offering valuable insights into users’ financial histories and relationships with other financial institutions before (and after) joining Nubank [6]. This helps us understand our customers more comprehensively and build solutions tailored to their needs, empowering them with better financial products.

Experimental validation

The inclusion of various data sources and their representations is rigorously tested through experimentation and empirical testing against offline evaluation metrics [6]. This iterative process reveals that simply adding more tokens doesn’t always improve performance; sometimes, due to context length limitations, it can even degrade it. Our framework allows us to rapidly go from a raw data source to test results in a couple of days, dramatically reducing the effort and guesswork involved in traditional feature engineering [6].

Measuring impact: Evaluating our pretrained models

Initial evaluations of our pretrained models involved training gradient-boosted trees on the resulting embeddings to make predictions. However, we found that this approach often captured limited signal and performance deltas did not always propagate to our downstream models. To truly unlock the potential of these models, we shifted towards fine-tuning them directly with labels for specific tasks [3].

Supervised fine-tuning

Our supervised fine-tuning process involves adding a linear prediction head to a pre-trained transformer. This head predicts a given label (binary, multi-class, or regression) from the final token embedding, also known as the user embedding.

To achieve optimal performance on downstream tasks, the finetuned model typically includes various task specific tabular features that are fed to a blending module that extracts information from those features and blends with transaction embedding information. Doing things this way can lead to incredible downstream task performance, but this evaluates the power of the combination of transaction embeddings and tabular features. 

To evaluate pretrained models alone, the pretrained backbone can be frozen, and the fine-tuned model can be fed only transactions, leaving the tabular features out of the blending head. This can help isolate the impact of the pretrained transaction embeddings to give some indication of the power of the pretrained model and its embeddings. Still, this requires finetuning, which requires additional time and computation to evaluate a pretrained model. In the future, we aim to create better benchmarks for standalone pretrained models that can create unsupervised embeddings for a variety of tasks.

An active research frontier

The world of foundation models in finance is a rapidly evolving area. While we’ve made incredible strides, this is an active area of research, and there’s much more to come! We are continually exploring new architectures, pretraining strategies, model use cases, and evaluation methodologies to push the boundaries of what’s possible.

The path forward

Our initial successes in deploying Predictive Foundation Models into Nubank’s AI Platform mark an early but significant step towards our AI-First vision [11]. Our approach remains consistent and ambitious:

• Ingest and validate more unstructured data sources.
• Train more advanced Foundation Models.
• Deploy them to more critical use cases.

We will continue to explore optimal model parameters, incorporate even more diverse data sources, and refine our evaluation techniques. This sweeping shift in technology and product thinking will further enable Nubank to understand its customers beyond the capabilities of existing methods, ultimately helping us meet their financial needs at the right time and empower them with truly personalized and intelligent financial solutions [3].

The future of finance is intelligent, and at Nubank, we are building it one insightful transaction at a time.

References

1. Braithwaite, D., & Udagawa, H. (2025a). Defining an interface between transaction data and foundation models – Building Nubank.
2. Braithwaite, D., Cavalcanti, M., & Udagawa, H. (2025b). Fine-tuning transaction user models – Building Nubank.
3. Braithwaite, D., & Udagawa, H. (2025c). Understanding our customers’ finances through foundation models – Building Nubank.
4. Braithwaite, D. T., Cavalcanti, M., McEver, R. A., Udagawa, H., Silva, D., Ramanath, R., Meneses, F., Yoshida, A., Wingert, E., Ramos, M., Zanfelice, B., & Gupta, A. (2025d). Your spending needs attention: Modeling financial habits with transformers. arXiv preprint arXiv:2507.23267.
5. Dao, T., Fu, D., Ermon, S., Rudra, A., & Ré, C. (2022). Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems (Vol. 35, pp. 16344–16359).
6. Foust, T. (2025). Optimizing user narratives for foundation models – Building Nubank.
7. Gorishniy, Y., Rubachev, I., & Babenko, A. (2022). On embeddings for numerical features in tabular deep learning. In Advances in Neural Information Processing Systems (Vol. 35, pp. 24991–25004).
8. Imrie, F., Denner, S., Brunschwig, L. S., Maier-Hein, K., & Van Der Schaar, M. (2025). Automated ensemble multimodal machine learning for healthcare. IEEE Journal of Biomedical and Health Informatics.
9. Kazemnejad, A., Padhi, I., Ramamurthy, K. N., Das, P., & Reddy, S. (2023). The impact of positional encoding on length generalization in transformers. In Advances in Neural Information Processing Systems (Vol. 36, pp. 24892–24928).
10. Li, J., Wang, M., Li, J., Fu, J., Shen, X., Shang, J., & McAuley, J. (2023). Text is all you need: Learning language representations for sequential recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1258–1267).
11. Udagawa, H. (2025). Building foundation models into Nubank’s AI platform – Building Nubank.
12. Wang, R., Shivanna, R., Cheng, D., Jain, S., Lin, D., Li, L., & Chi, E. H. (2021). Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the Web Conference 2021 (pp. 1785–1797).