Avalanche stack and real-time streaming applications at Nu

The topic debated at the Nubank DS&ML Meetup #98 was the transformative power of real-time data processing in digital banking. At Nubank, our proprietary Avalanche stack integrates advanced technologies to enable instant insights and rapid responses. 

From robust fraud detection to personalized customer interactions, explore how our machine learning engineers—Caroline Custódio, Lucas Nolasco, Luiz Felix, and Otávio Vasques—discuss how Nu is revolutionizing security, efficiency, and user experience through cutting-edge real-time analytics.

The Avalanche stack: Powering real-time data processing at Nubank

Avalanche is a standardized approach to real-time streaming data that integrates smoothly into Nubank’s architecture. While traditional batch processing analyzes data periodically, Avalanche processes data streams, allowing Nubank to identify trends and anomalies in real-time.

This capability enhances decision-making across multiple teams at Nu.

Core components of Avalanche

Kubernetes: The backbone of streaming infrastructure

Kubernetes efficiently orchestrates resource allocation and application management essential for streaming operations at scale. Its custom resource definitions and automated deployment processes streamline the maintenance of complex streaming workflows. 

At Nubank, Kubernetes serves as the foundation for deploying Apache Flink applications, ensuring scalability and reliability under high-throughput conditions.

Apache Kafka: Robust messaging and buffering system

Kafka serves as the dependable messaging layer within Avalanche, facilitating reliable, fault-tolerant communication between streaming applications. 

It retains messages temporarily, allowing recovery from failures without data loss, thus providing consistency and ensuring continuous system availability, even in high-demand situations.

Apache Flink: Real-time data processing framework

Flink delivers powerful capabilities for real-time data analytics, supporting sophisticated tasks such as event-time windowing, stateful stream processing, and real-time data joins. 

It adeptly manages both bounded (finite) and unbounded (infinite) data streams, enabling Nubank to implement advanced analytics and sophisticated business logic dynamically and efficiently.

Apache Pinot: Analytical database for user-facing real-time queries

Pinot complements Flink by offering a highly responsive analytical database specifically designed for real-time user interactions. 

It empowers Nubank’s analysts and developers to execute complex aggregations and queries instantaneously, significantly enhancing the speed and flexibility of real-time data-driven decisions.

Applications in fraud detection

Real-time sequential models for fraud prevention

Sequential neural networks are leveraged to detect fraud in real-time. These models process recent customer transaction data and behavioral patterns within milliseconds, enabling highly accurate fraud risk assessments.

By integrating Flink and Pinot, the system continuously analyzes new and historical events, rapidly identifying suspicious activities. For instance, when a potentially fraudulent transaction occurs, immediate analysis allows proactive intervention, significantly reducing the risk of financial loss and enhancing customer trust.

Autopilot: Real-time defense calibration

Autopilot was introduced as a system for dynamically adjusting anti-fraud measures based on real-time data. Static thresholds for fraud detection can quickly become outdated due to evolving patterns of fraudulent behavior. Autopilot monitors incoming transaction streams continuously, recalibrating detection thresholds automatically and incrementally.

Using Avalanche’s real-time analytics capabilities, Autopilot rapidly adapts to changes, proactively managing fraud risk. This agile approach not only optimizes the use of operational resources and maintains SLA control but also provides rapid response times, essential in digital finance.

On-Demand Features Handler: Simplifying real-time feature management

The On-Demand Features Handler was created to simplify real-time feature management across numerous data-driven applications. By standardizing the production and consumption of data features, engineers can focus more effectively on developing high-impact machine learning models rather than complex infrastructure details.

This handler provides an intuitive HTTP interface, abstracting intricate backend processes involving Kafka, Flink, and Pinot. The approach enhances collaboration, ensures data consistency, and accelerates the deployment of reliable, high-performing machine learning solutions.

Case study: Preventing opportunistic loans

A practical implementation of this system involves detecting opportunistic loan applicants: those unlikely to repay. By analyzing real-time customer interactions within the Nubank app, machine learning models accurately identify risky loan behaviors.

This application achieved a 13% improvement in the model’s effectiveness at identifying high-risk applicants compared to previous systems, demonstrating the tangible benefits of Avalanche’s real-time analytics capabilities. Such an improvement translates directly into better financial risk management, safeguarding Nubank and its customers.

Real-time analytics via the Avalanche stack empowers Nubank to lead digital financial innovation. By harnessing real-time insights to drive fraud prevention, customer engagement, and operational efficiency, Nubank sets new benchmarks for responsiveness and effectiveness in the financial services industry. 

Avalanche’s advanced streaming capabilities continue to position Nubank at the forefront, shaping the future of banking through innovation and technological excellence.