Browsing Category

Apache Spark

10 posts
Shift Left Architecture with Confluent Data Streaming and Databricks Lakehouse Medallion
Read More
  • 833 views
  • 9 minute read

Shift Left Architecture for AI and Analytics with Confluent and Databricks

Confluent and Databricks enable a modern data architecture that unifies real-time streaming and lakehouse analytics. By combining shift-left principles with the structured layers of the Medallion Architecture, teams can improve data quality, reduce pipeline complexity, and accelerate insights for both operational and analytical workloads. Technologies like Apache Kafka, Flink, and Delta Lake form the backbone of scalable, AI-ready pipelines across cloud and hybrid environments.
Read More
0
0
0
Confluent and Databricks for Data Integration and Stream Processing
Read More
  • 1.0K views
  • 10 minute read

Confluent Data Streaming Platform vs. Databricks Data Intelligence Platform for Data Integration and Processing

This blog explores how Confluent and Databricks address data integration and processing in modern architectures. Confluent provides real-time, event-driven pipelines connecting operational systems, APIs, and batch sources with consistent, governed data flows. Databricks specializes in large-scale batch processing, data enrichment, and AI model development. Together, they offer a unified approach that bridges operational and analytical workloads. Key topics include ingestion patterns, the role of Tableflow, the shift-left architecture for earlier data validation, and real-world examples like Uniper’s energy trading platform powered by Confluent and Databricks.
Read More
0
0
0
Fraud Prevention in Mobility Services with Data Streaming using Apache Kafka and Flink with AI Machine Learning
Read More
  • 1.3K views
  • 7 minute read

Fraud Detection in Mobility Services (Ride-Hailing, Food Delivery) with Data Streaming using Apache Kafka and Flink

Mobility services like Uber, Grab, and FREE NOW (Lyft) rely on real-time data to power seamless trips, deliveries, and payments. But this real-time nature also opens the door to sophisticated fraud schemes—ranging from GPS spoofing to payment abuse and fake accounts. Traditional fraud detection methods fall short in speed and adaptability. By using Apache Kafka and Apache Flink, leading mobility platforms now detect and block fraud as it happens, protecting their revenue, users, and trust. This blog explores how real-time data streaming is transforming fraud prevention across the mobility industry.
Read More
0
0
0
The Data Streaming Landscape 2025 with Kafka Flink Confluent Amazon MSK Cloudera Event Hubs and Other Platforms
Read More
  • 9.0K views
  • 21 minute read

The Data Streaming Landscape 2025

Data streaming is a new software category. It has grown from niche adoption to becoming a fundamental part of modern data architecture, leveraging open source technologies like Apache Kafka and Flink. With real-time data processing transforming industries, the ecosystem of tools, platforms, and cloud services has evolved significantly. This blog post explores the data streaming landscape of 2025, analyzing key players, trends, and market dynamics shaping this space.
Read More
0
0
1
Data Streaming Trends for 2025 - Leading with Apache Kafka and Flink
Read More
  • 12.2K views
  • 18 minute read

Top Trends for Data Streaming with Apache Kafka and Flink in 2025

Apache Kafka and Apache Flink are leading open-source frameworks for data streaming that serve as the foundation for cloud services, enabling organizations to unlock the potential of real-time data. Over recent years, trends have shifted from batch-based data processing to real-time analytics, scalable cloud-native architectures, and improved data governance powered by these technologies. Looking ahead to 2025, the data streaming ecosystem is set to undergo even greater changes. Here are the top trends shaping the future of data streaming for businesses.
Read More
0
0
0
Data Streaming Landscape 2024 around Kafka Flink and Cloud
Read More
  • 13.8K views
  • 21 minute read

The Data Streaming Landscape 2024

The research company Forrester defines data streaming platforms as a new software category in a new Forrester Wave. Apache Kafka is the de facto standard used by over 100,000 organizations. Plenty of vendors offer Kafka platforms and cloud services. Many complementary open source stream processing frameworks like Apache Flink and related cloud offerings emerged. And competitive technologies like Pulsar, Redpanda, or WarpStream try to get market share leveraging the Kafka protocol. This blog post explores the data streaming landscape of 2024 to summarize existing solutions and market trends. The end of the article gives an outlook to potential new entrants in 2025.
Read More
0
0
3
Data Streaming Landscape 2023 with Apache Kafka Flink and much more
Read More
  • 9.5K views
  • 13 minute read

The Data Streaming Landscape 2023

Data streaming is a new software category to process data in motion. Apache Kafka is the de facto standard used by over 100,000 organizations. Plenty of vendors offer Kafka platforms and cloud services. Many complementary stream processing engines like Apache Flink and SaaS offerings have emerged. And competitive technologies like Pulsar and Redpanda try to get market share. This blog post explores the data streaming landscape of 2023 to summarize existing solutions and market trends.
Read More
1
0
0
Case Studies for Cloud Native Analytics with Data Warehouse Data Lake Data Streaming Lakehouse
Read More
  • 8.7K views
  • 7 minute read

Case Studies: Cloud-native Data Streaming for Data Warehouse Modernization

The concepts and architectures of a data warehouse, a data lake, and data streaming are complementary to solving business problems. Unfortunately, the underlying technologies are often misunderstood, overused for monolithic and inflexible architectures, and pitched for wrong use cases by vendors. Let’s explore this dilemma in a blog series. This is part 4: Case Studies for cloud-native data streaming and data warehouses.
Read More
43
0
0

Apache Kafka Streams + Machine Learning (Spark, TensorFlow, H2O.ai)

Apache Kafka Streams to build Real Time Streaming Microservices. Apply Machine Learning / Deep Learning using Spark, TensorFlow, H2O.ai, etc. to add AI. Embed Kafka Streams into Java App, Docker, Kubernetes, Mesos, anything else.
Read More
0
0
0