Tagged | distributed-systems
-
FOQS: Making a distributed priority queue disaster-ready
(engineering.fb.com)#software-engineering #software-architecture #infra #distributed-systems
-
Message Queuing and the Database: Solving the Dual Write Problem
(www.cockroachlabs.com) -
A Lightweight Distributed Architecture to Handle Thousands of Library Releases at eBay
(tech.ebayinc.com)#software-engineering #software-architecture #distributed-systems
-
Improving Reliability: Building a Vitess Balancer to Minimize MySQL Downtime
(product.hubspot.com) -
Pinterest Druid Holiday Load Testing
(medium.com) -
eBay’s Global Secondary Indexes
(tech.ebayinc.com) -
Change Data Capture: Fine Tuning Changefeeds for Performance and Durability
(www.cockroachlabs.com) -
How Dropbox Replay keeps everyone in sync
(dropbox.tech)#software-engineering #software-architecture #distributed-systems
-
MemQ: An Efficient, Scalable Cloud Native PubSub System
(stackshare.io)#software-architecture #scaling #distributed-systems #cloud #pubsub
-
MemQ: An efficient, scalable cloud native PubSub system
(medium.com)#software-architecture #scaling #distributed-systems #data-engineering
-
Scaling Apache Druid for Real-Time Cloud Analytics at Confluent
(www.confluent.io) -
Presentation: Building and Scaling a Control Plane for 1000s of Kafka Clusters
(www.infoq.com) -
Database Sharding vs. Partitioning: What’s the Difference?
(www.singlestore.com) -
How do CRDTs solve distributed data consistency challenges?
(ably.com) -
CarbonJ: A high performance, high-scale, drop-in replacement for carbon-cache and carbon-relay
(engineering.salesforce.com)#software-architecture #scaling #distributed-systems #data-engineering
-
Efficient Resource Management at Pinterest’s Batch Processing Platform
(stackshare.io) -
Introducing uGroup: Uber’s Consumer Management Framework
(eng.uber.com)#software-architecture #distributed-systems #data-engineering
-
Processing billions of events in real time at Twitter
(blog.twitter.com)#software-architecture #scaling #distributed-systems #data-engineering
-
Tiny Letter from Kafka
(developers.soundcloud.com)#software-architecture #distributed-systems #apache-kafka #backend
-
Improving HDFS I/O Utilization for Efficiency
(eng.uber.com)#performance #distributed-systems #big-data #data-engineering
-
Evolution of Region Assignment in the Apache HBase Architecture — Part 3
(engineering.salesforce.com)#performance #scaling #distributed-systems #data-engineering
-
Article: A Lightweight, Safe, Portable, and High-Performance Runtime for Dapr
(www.infoq.com) -
Building Uber’s Fulfillment Platform for Planet-Scale using Google Cloud Spanner
(eng.uber.com) -
Groot: eBay’s Event-graph-based Approach for Root Cause Analysis
(tech.ebayinc.com) -
The Airflow Smart Sensor Service
(medium.com) -
Ensuring High Availability of Ads Realtime Streaming Services
(medium.com) -
Distributed tier merge: How LinkedIn tackles stragglers in search index build
(engineering.linkedin.com) -
Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot
(eng.uber.com)#software-architecture #distributed-systems #data-engineering
-
How Airbnb Enables Consistent Data Consumption at Scale
(medium.com)#data-pipeline #software-architecture #scaling #distributed-systems
-
Nrtsearch: Yelp’s Fast, Scalable and Cost Effective Search Engine
(engineeringblog.yelp.com)#software-architecture #search #scaling #distributed-systems
-
When is Stale Data OK? An Epic Read on Follower Reads
(www.cockroachlabs.com) -
Scaling LinkedIn's Hadoop YARN cluster beyond 10,000 nodes
(engineering.linkedin.com) -
Jellyfish: Cost-Effective Data Tiering for Uber’s Largest Storage System
(eng.uber.com)#software-architecture #performance #distributed-systems #data-engineering
-
Pinterest’s Analytics as a Platform on Druid (Part 3 of 3)
(medium.com) -
Enabling Seamless Kafka Async Queuing with Consumer Proxy
(eng.uber.com)#data-pipeline #software-architecture #distributed-systems #apache-kafka
-
Building Scalable Streaming Pipelines for Near Real-Time Features
(eng.uber.com) -
RAMP-TAO: Layering atomic transactions on Facebook’s online graph store
(engineering.fb.com) -
Logging at Twitter: Updated
(blog.twitter.com)#software-engineering #software-architecture #logging #distributed-systems
-
Efficiently Managing the Supply and Demand on Uber’s Big Data Platform
(eng.uber.com)#software-architecture #infra #distributed-systems #big-data
-
Cost-Efficient Open Source Big Data Platform at Uber
(eng.uber.com)#optimisation #distributed-systems #big-data #data-engineering
-
How we built a general purpose key value store for Facebook with ZippyDB
(engineering.fb.com) -
Unified Flink Source at Pinterest: Streaming Data Processing
(stackshare.io) -
Podcast: Michael Perry on Immutable Architecture, CAP Theorem, and CRDTs
(www.infoq.com) -
Containerizing Apache Hadoop Infrastructure at Uber
(eng.uber.com) -
Article: Building Latency Sensitive User Facing Analytics via Apache Pinot
(www.infoq.com)#distributed-systems #analytics #real-time #data-engineering
-
Efficient and Reliable Compute Cluster Management at Scale
(eng.uber.com) -
Navigating the 8 fallacies of distributed computing
(ably.com) -
Presentation: Essential Complexity in Systems Architecture
(www.infoq.com) -
Building Logging for a Distributed Database: Splunk vs ELK vs BYO
(www.cockroachlabs.com) -
How We Built REGIONAL BY ROW for Row-Level Data Homing & Distribution
(www.cockroachlabs.com) -
Block Aggregator: Real-time Data Ingestion from Kafka to ClickHouse with Deterministic Retries
(tech.ebayinc.com) -
How to Deploy a Multi-Region Application: Under the Hood
(www.cockroachlabs.com) -
The exabyte club: LinkedIn’s journey of scaling the Hadoop Distributed File System
(engineering.linkedin.com)#scaling #distributed-systems #analytics #big-data #data-engineering
-
The Mysterious Gotcha of gRPC Stream Performance
(ably.com)#debugging #performance #distributed-systems #networking #RPC
-
Jolie - A Service-Oriented Programming Language for Distributed Applications
(www.infoq.com) -
Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions
(engineering.grab.com)#data-pipeline #infra #machine-learning #scaling #distributed-systems
-
Using kafka-merge-purge to Deal with Failure in an Event-Driven System at FLYERALARM
(www.confluent.io) -
Presentation: Change Data Capture for Distributed Databases @Netflix
(www.infoq.com) -
Netflix Drive
(netflixtechblog.com) -
How Airbnb Achieved Metric Consistency at Scale
(medium.com)#software-architecture #distributed-systems #analytics #data-engineering
-
Google Provides a Peek into the Architecture of Colossus - Its Storage Foundation
(www.infoq.com) -
Scaling Kubernetes with Assurance at Pinterest
(stackshare.io) -
Dependable realtime banking with Kafka and Ably
(ably.com) -
Presentation: Solving Mysteries Faster with Observability
(www.infoq.com) -
Load Testing with Koi Pond
(slack.engineering)#software-engineering #software-architecture #testing #distributed-systems
-
On Coordinated Omission
(www.scylladb.com) -
Adopting RocksDB within Manhattan
(blog.twitter.com) -
Attack of the Delta Clones (Against Disaster Recovery Availability Complexity)
(databricks.com)#software-architecture #infra #distributed-systems #data-engineering
-
FullContact: Improving the Graph by Transitioning to Scylla
(www.scylladb.com) -
Couchbase Details Its Distributed ACID Transaction Architecture
(www.infoq.com) -
Load Balancing in Scylla Alternator
(www.scylladb.com) -
The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 2
(engineering.salesforce.com) -
The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 1
(engineering.salesforce.com) -
Testing Apicurio Registry’s performance and scalability
(developers.redhat.com) -
Presentation: The Medieval Census Problem
(www.infoq.com) -
Article: The Evolution of Distributed Systems on Kubernetes
(www.infoq.com) -
Uber Implements Disaster Recovery for Multi-Region Kafka
(www.infoq.com)#software-architecture #infra #distributed-systems #apache-kafka
-
One billion files in Ozone
(blog.cloudera.com) -
Turbine: Facebook’s service management platform for stream processing
(engineering.fb.com) -
Patterns for resilient distributed systems
(levelup.gitconnected.com) -
Presentation: Scalable, Cloud-native Data Applications by Example
(www.infoq.com) -
Preventing Fraud and Fighting Account Takeovers with Kafka Streams
(www.confluent.io)#data-pipeline #software-architecture #security #distributed-systems
-
Multi-zone Cluster Management at Wayfair with Kubernetes
(tech.wayfair.com) -
Presentation: Streaming a Million likes/second: Real-time Interactions on Live Video
(www.infoq.com)#software-architecture #scaling #distributed-systems #data-engineering
-
Distribute your microservices data with events, CQRS, and event sourcing
(developers.redhat.com) -
Introducing Quicksilver: Configuration Distribution at Internet Scale
(blog.cloudflare.com)#software-architecture #infra #distributed-systems #internet
-
Advanced schema management for Spark applications at scale
(engineering.linkedin.com) -
Edge to Cloud: Building the Tesla Virtual Power Plant
(www.infoq.com) -
InSearch: LinkedIn’s new message search platform
(engineering.linkedin.com)#software-architecture #search #scaling #distributed-systems
-
Enabling HDFS Federation Having 1B File System Objects
(tech.ebayinc.com) -
Hash Sharded Indexes Unlock Linear Scaling for Sequential Workloads
(www.cockroachlabs.com) -
Presentation: "This Website is Not Secured" You Had One Job: Configuring the Edge Proxy!
(www.infoq.com) -
How we improved latency through projection in Espresso
(engineering.linkedin.com)#software-architecture #DBMS #distributed-systems #data-engineering
-
Presentation: Kafka Needs No Keeper
(www.infoq.com)#software-architecture #distributed-systems #apache-kafka #backend
-
How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience
(netflixtechblog.com)#DBMS #distributed-systems #analytics #real-time #data-engineering
-
Supporting Spark as a First-Class Citizen in Yelp’s Computing Platform
(engineeringblog.yelp.com)#data-pipeline #distributed-systems #apache-spark #big-data #backend
-
Running Online Services at Riot: Part VI
(technology.riotgames.com) -
99th Percentile Latency at Scale with Apache Kafka
(www.confluent.io) -
Nauto: Achieving Consistency in an Eventually Consistent Environment
(www.scylladb.com) -
Building an Adaptive, Multi-Tenant Stream Bus with Kafka and Golang
(eng.lyft.com) -
Debugging Distributed Systems: 3 Common Distributed Tracing Challenges & How to Overcome Them
(blog.overops.com) -
Article: Anatomy of Cascading Failure
(www.infoq.com) -
RabbitMQ Quorum Queues explained - what you need to know.
(erlang-solutions.com)#software-architecture #distributed-systems #event-queue #backend
-
Design for Failure — Distributed Transaction in Microservices
(levelup.gitconnected.com) -
WKSctl: A Tool for Kubernetes Cluster Management Using GitOps
(www.infoq.com) -
Presentation: Snowflake Architecture: Building a Data Warehouse for the Cloud
(www.infoq.com) -
How Twitter Improves Resource Usage With a Deterministic Load Balancing Algorithm
(www.infoq.com) -
Streaming Machine Learning with Tiered Storage and Without a Data Lake
(www.confluent.io)#data-pipeline #machine-learning #distributed-systems #apache-kafka
-
Infinite Storage in Confluent Platform
(www.confluent.io)#distributed-systems #apache-kafka #big-data #data-engineering
-
Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing
(eng.uber.com)#data-pipeline #software-architecture #distributed-systems #data-engineering
-
Stop the Insanity: Eliminating Data Infrastructure Sprawl
(www.memsql.com) -
Presentation: Fault Tolerance at Speed
(www.infoq.com)#software-architecture #performance #scaling #distributed-systems
-
Streams and Tables in Apache Kafka: Elasticity, Fault Tolerance, and Other Advanced Concepts
(www.confluent.io) -
Engineering SQL Support on Apache Pinot at Uber
(eng.uber.com) -
Streams and Tables in Apache Kafka: Processing Fundamentals
(www.confluent.io) -
Reliably Upgrading Apache Airflow at Slack’s Scale
(slack.engineering) -
Comcast: Sprinting from Cassandra to Scylla
(www.scylladb.com)#software-architecture #performance #distributed-systems #data-engineering
-
Streams and Tables in Apache Kafka: Topics, Partitions, and Storage Fundamentals
(www.confluent.io) -
Pipeline to the Cloud – Streaming On-Premises Data for Cloud Analytics
(www.confluent.io)#data-pipeline #distributed-systems #apache-kafka #analytics
-
Plumbing At Scale
(engineering.grab.com)#data-pipeline #software-architecture #scaling #distributed-systems
-
Routing Engine — Central Work Dispenser
(medium.com)#software-architecture #scaling #distributed-systems #backend
-
Presentation: Beyond Microservices: Streams, State and Scalability
(www.infoq.com)#software-architecture #scaling #distributed-systems #microservices #backend
-
Deterministic Aperture: A distributed, load balancing algorithm
(blog.twitter.com) -
Apache Kafka Producer Improvements with the Sticky Partitioner
(www.confluent.io)#software-architecture #performance #distributed-systems #apache-kafka
-
Streaming Cassandra into Kafka in (Near) Real-Time: Part 2
(engineeringblog.yelp.com)#data-pipeline #distributed-systems #real-time #data-engineering
-
Presentation: Concurrency, Scalability and Transactions -- Myths and Surprises
(www.infoq.com)#software-architecture #scaling #distributed-systems #concurrency
-
Uber’s Data Platform in 2019: Transforming Information to Intelligence
(eng.uber.com)#data-pipeline #scaling #distributed-systems #data-engineering
-
Presentation: Monoliths, Microservices, Events, Functions: What It Takes to Go Through the Transformation
(www.infoq.com)#software-architecture #distributed-systems #microservices #backend
-
How Scylla Scaled to One Billion Rows a Second
(www.scylladb.com) -
Dynein: Building a Distributed Delayed Job Queueing System
(medium.com) -
Presentation: Building Reactive Pipelines: How to Go from Scalable Apps to (Ridiculously) Scalable Systems
(www.infoq.com) -
Streaming Cassandra into Kafka in (Near) Real-Time: Part 1
(engineeringblog.yelp.com)#data-pipeline #software-architecture #distributed-systems #data-engineering
-
Kubernetes the Very Hard Way With Large Clusters at Datadog
(www.infoq.com)#software-architecture #scaling #distributed-systems #kubernetes #backend
-
Reducing Multi-Region Latency with Follower Reads
(www.cockroachlabs.com) -
Building Large Kubernetes Clusters
(engineering.linecorp.com)#software-engineering #distributed-systems #microservices #kubernetes
-
How Shopify Implements Custom Autoscaling Rules in Kubernetes
(www.infoq.com) -
Optimizing Observability with Jaeger, M3, and XYS at Uber
(eng.uber.com) -
Availability and Region Failure: Joint Consensus in CockroachDB
(www.cockroachlabs.com)#DBMS #scaling #distributed-systems #reliability #data-engineering
-
Egnyte Architecture: Lessons learned in building and scaling a multi petabyte content platform
(highscalability.com)#software-architecture #infra #scaling #distributed-systems #internet
-
Getting Started with Rust and Apache Kafka
(www.confluent.io) -
Lessons Learned Building Distributed Systems with CQRS and Event Sourcing
(hackernoon.com)#software-architecture #distributed-systems #microservices #backend
-
Maximizing Performance via Concurrency While Minimizing Timeouts in Distributed Databases
(www.scylladb.com)#DBMS #performance #distributed-systems #concurrency #data-engineering
-
Evolution of Zulily’s Airflow Infrastructure
(zulily-tech.com) -
Presentation: The Future of Chaos Engineering: In Pursuit of the Unknown Unknowns
(www.infoq.com) -
Remember Clusterman? Now It's Open-Source, and Supports Kubernetes Too!
(engineeringblog.yelp.com) -
Conducting Better Business with Uber’s Open Source Orchestration Tool, Cadence
(eng.uber.com)#software-architecture #distributed-systems #microservices #backend
-
Parallel Commits: An Atomic Commit Protocol For Globally Distributed Transactions
(www.cockroachlabs.com) -
Fast dimensional analysis for root cause analysis at scale
(engineering.fb.com) -
Achieving optimal query performance with a distributed time-series database on PostgreSQL
(blog.timescale.com) -
Optimizing Search Index Generation using secondary cache
(medium.com)#performance #distributed-systems #big-data #caching #data-engineering
-
GRIT Protocol Enables Distributed Transactions across Multi-Database Microservices
(www.infoq.com) -
Business Continuity & Disaster Recovery in the Microservices world
(medium.com)#software-architecture #distributed-systems #microservices #backend
-
RabbitMQ Mirrored Queues Gotchas
(erlang-solutions.com) -
When you deserve better (systems)
(tech.gc.com)#software-engineering #software-architecture #infra #distributed-systems
-
GRIT: a Protocol for Distributed Transactions across Microservices
(tech.ebayinc.com)#software-architecture #distributed-systems #microservices #protocols
-
Securing Apache Airflow UI With DAG Level Access
(eng.lyft.com) -
Presentation: High Performance Cooperative Distributed Systems in Adtech
(www.infoq.com)#software-architecture #performance #scaling #distributed-systems
-
🚂 On Track with Apache Kafka – Building a Streaming ETL Solution with Rail Data
(www.confluent.io) -
Delta: A Data Synchronization and Enrichment Platform
(medium.com)#software-architecture #algorithms #distributed-systems #data-engineering
-
How LinkedIn customizes Apache Kafka for 7 trillion messages per day
(engineering.linkedin.com)#performance #scaling #distributed-systems #apache-kafka #data-engineering
-
Real-time experiment analytics at Pinterest using Apache Flink
(medium.com) -
Scribe: Transporting petabytes per hour via a distributed, buffered queueing system
(engineering.fb.com)#software-architecture #scaling #distributed-systems #networking
-
Building a Service Mesh with Envoy
(www.thoughtworks.com) -
Presentation: Peloton - Uber's Webscale Unified Scheduler on Mesos & Kubernetes
(www.infoq.com) -
Scaling a Mature Data Pipeline — Managing Overhead
(medium.com)#data-pipeline #software-architecture #scaling #distributed-systems
-
Incremental Cooperative Rebalancing in Apache Kafka: Why Stop the World When You Can Change It?
(www.confluent.io) -
Evolving Regional Evacuation
(medium.com) -
Shared Transactional Tables: The Foundation of Next Generation Big Data Warehousing
(blog.cloudera.com) -
How Sqreen handles 50,000 requests every minute in a write-heavy environment
(stackshare.io) -
Solving manageability challenges at scale with Nuage
(engineering.linkedin.com) -
A Shallow Dive Into Distributed Tracing
(kinvolk.io) -
A look inside Kafka Mirrormaker 2
(blog.cloudera.com) -
Presentation: CockroachDB: Architecture of a Geo-distributed SQL Database
(www.infoq.com)#software-architecture #DBMS #algorithms #distributed-systems #data-engineering
-
Presentation: Conquering Microservices Complexity @Uber with Distributed Tracing
(www.infoq.com)#software-architecture #scaling #distributed-systems #microservices
-
A Technical Introduction to MemSQL
(www.memsql.com)#software-architecture #DBMS #distributed-systems #data-engineering
-
Podcast: Yuri Shkuro on Tracing Distributed Systems Using Jaeger
(www.infoq.com) -
Presto Infrastructure at Lyft
(eng.lyft.com)#infra #scaling #distributed-systems #backend #data-engineering
-
SQS FIFO Queues: Message Ordering and Exactly-Once Processing Guaranteed?
(www.ably.io)#software-design #distributed-systems #event-queue #cloud #systems
-
Building a Fault-Tolerant Data Pipeline for Chatbots
(engineering.salesforce.com)#data-pipeline #software-engineering #software-architecture #distributed-systems #backend
-
Building Transactional Systems Using Apache Kafka
(www.confluent.io) -
SQL Prober: Black-box Monitoring in Managed CockroachDB
(www.cockroachlabs.com) -
NuRaft: a Lightweight C++ Raft Core
(tech.ebayinc.com) -
Our Journey to Optimal Job Sizes for Apache Spark
(engineering.salesforce.com)#data-pipeline #software-architecture #distributed-systems #apache-spark #backend
-
Building Shared State Microservices for Distributed Systems Using Kafka Streams
(www.confluent.io)#software-architecture #distributed-systems #microservices #apache-kafka #backend
-
Evolution of Netflix Conductor:
(medium.com) -
YuniKorn: a universal resources scheduler
(blog.cloudera.com)#software-architecture #distributed-systems #microservices #backend #scheduler
-
Presentation: PID Loops and the Art of Keeping Systems Stable
(www.infoq.com)#software-design #software-architecture #algorithms #distributed-systems #systems
-
Fault Tolerance in Distributed Systems: Tracing with Apache Kafka and Jaeger
(www.confluent.io) -
Presentation: Streaming Log Analytics with Kafka
(www.infoq.com) -
Fast Parallel Testing at Databricks with Bazel
(databricks.com) -
Presentation: Automatic Clustering at Snowflake
(www.infoq.com)#infra #DBMS #scaling #distributed-systems #data-engineering
-
Making Apache Spark Effortless for All of Uber
(eng.uber.com)#software-architecture #DBMS #distributed-systems #apache-spark #data-engineering
-
YuniKorn: a universal resource scheduler
(blog.cloudera.com)#software-architecture #distributed-systems #orchestration #backend
-
Open Sourcing Brooklin: Near Real-Time Data Streaming at Scale
(engineering.linkedin.com)#data-pipeline #software-architecture #scaling #distributed-systems #data-engineering
-
Building distributed apps: second approximation
(towardsdatascience.com) -
OIL+VCache: File abstraction for distributed systems
(code.fb.com) -
Patterns in Distributed Systems
(www.infoq.com) -
Presentation: Verifying a Distributed System with Combinatorial Topology
(www.infoq.com)#software-architecture #scaling #distributed-systems #kubernetes
-
Kafka Listeners – Explained
(www.confluent.io)#software-architecture #distributed-systems #apache-kafka #internals #data-engineering
-
Improving Performance and Capacity for Espresso with New Netty Framework
(engineering.linkedin.com)#software-architecture #performance #distributed-systems #data-engineering
-
Presentation: Unique Resiliency of the BEAM and Erlang OTP
(www.infoq.com)#scaling #distributed-systems #runtime #availability #erlang
-
Presentation: Using Randomized Communication for Robust, Scalable Systems
(www.infoq.com) -
Distributed Deep Learning Pipelines with PySpark and Keras
(towardsdatascience.com) -
High Availability Kafka Service in Microsoft Azure Cloud
(medium.com) -
Star-Tree Index: Powering Fast Aggregations on Pinot
(engineering.linkedin.com) -
Presentation: Reactive Systems' Architecture
(www.infoq.com) -
Streaming Data from the Universe with Apache Kafka
(www.confluent.io)#software-architecture #distributed-systems #apache-kafka #data-engineering
-
HDFS Erasure Coding in Production
(blog.cloudera.com) -
Presentation: People You May Know: Fast Recommendations Over Massive Data
(www.infoq.com)#performance #distributed-systems #real-time #graphDB #data-engineering
-
Bringing scalable real-time analytics to the enterprise
(www.oreilly.com)#DBMS #scaling #distributed-systems #podcast #data-engineering
-
Presentation: Michelangelo Palette: A Feature Engineering Platform at Uber
(www.infoq.com)#data-science #machine-learning #distributed-systems #data-engineering
-
Presentation: Life of a Distributed Graph Database Query
(www.infoq.com)#DBMS #distributed-systems #graph-processing #data-engineering
-
An Overview of Sharding in PostgreSQL and How it Relates to MongoDB’s
(www.percona.com) -
Observability with the Elastic Stack
(stackshare.io) -
Schemas, Contracts, and Compatibility
(www.confluent.io) -
Presentation: CRDTs: From Sequential to Concurrent Executions
(www.infoq.com)#algorithms #distributed-systems #synchronisation #decentralised
-
Presentation: Docker Data Science Pipeline
(www.infoq.com)#data-pipeline #software-architecture #docker #distributed-systems #hadoop
-
10x Faster Parallel Python Without Python Multiprocessing
(towardsdatascience.com)#performance #distributed-systems #python #parallel-computing
-
Kafka Replication: The case for MirrorMaker 2.0
(blog.cloudera.com) -
Automatic Table Statistics in CockroachDB
(www.cockroachlabs.com) -
Small Files, Big Foils: Addressing the Associated Metadata and Application Challenges
(blog.cloudera.com)#software-architecture #distributed-systems #hadoop #systems
-
Back-Pressure Strategy for a Sharded Akka Cluster
(jobs.zalando.com) -
Apache Kafka Data Access Semantics: Consumers and Membership
(www.confluent.io) -
Partition Management in Hadoop
(blog.cloudera.com) -
How we optimized Magic Pocket for cold storage
(blogs.dropbox.com)#software-architecture #distributed-systems #systems #storage
-
Real-time data processing for monitoring and reporting — A practical use case of spark structured…
(medium.com)#data-pipeline #stream-processing #distributed-systems #apache-spark #data-engineering
-
Beam: A Distributed Knowledge Graph Store
(www.ebayinc.com)#DBMS #distributed-systems #GoLang #semantic-data #data-engineering
-
Presentation: YugaByte DB - A Planet-scale Database for Low Latency Transactional Apps
(www.infoq.com)#DBMS #performance #scaling #distributed-systems #data-engineering
-
Optimizing Kafka Streams Applications
(www.confluent.io)#performance #optimisation #distributed-systems #apache-kafka #backend
-
Understanding Kafka with Factorio
(hackernoon.com) -
Presentation: Designing Distributed Systems with TLA+
(www.infoq.com) -
Distributed SQL: An Evolution of the Database
(www.cockroachlabs.com) -
Troubleshooting Data Engineering Software
(engineering.linecorp.com)#debugging #performance #distributed-systems #backend #data-engineering
-
Consistent Data Partitioning through Global Indexing for Large Apache Hadoop Tables at Uber
(eng.uber.com) -
Avoiding Double Payments in a Distributed Payments System
(medium.com) -
Presentation: Interaction Protocols: It's All about Good Manners
(www.infoq.com)#software-engineering #API #distributed-systems #history #protocols
-
How eBay Governs its Big Data Fabric
(www.ebayinc.com) -
Presentation: Complex Event Flows in Distributed Systems
(www.infoq.com) -
ex_rabbit_pool open source AMQP connection pool
(erlang-solutions.com) -
Scale with Speed: The Bitcoin Lightning Network Explained
(www.toptal.com) -
Presentation: Cultivating Production Excellence - Taming Complex Distributed Systems
(www.infoq.com)#monitoring #distributed-systems #practices #backend #production
-
Troubleshooting Data Engineering Software
(engineering.linecorp.com)#debugging #performance #distributed-systems #data-engineering
-
Reliable, Distributed Locking in the Cloud
(tech.showmax.com) -
Multi-Segment Distributed Storage for Kubernetes
(itnext.io) -
Uber Case Study: Choosing the Right HDFS File Format for Your Apache Spark Jobs
(eng.uber.com) -
Solving Big Data Challenges with Data Science at Uber
(eng.uber.com)#DBMS #scaling #distributed-systems #big-data #data-engineering
-
Kafka Streams’ Take on Watermarks and Triggers
(www.confluent.io)#data-pipeline #stream-processing #distributed-systems #apache-kafka
-
Rabbit’s Anatomy - Understanding Topic Exchanges
(erlang-solutions.com) -
How we moved a massively parallel Postgres database onto Kubernetes
(engineering.pivotal.io) -
Presentation: Towards Language Support for Distributed Systems
(www.infoq.com)#programming-languages #distributed-systems #serverless #programming-paradigm
-
DBEvents: A Standardized Framework for Efficiently Ingesting Data into Uber’s Apache Hadoop Data Lake
(eng.uber.com)#data-pipeline #distributed-systems #hadoop #data-engineering
-
Stretching the Mold: How Alibaba Enhances Database Flexibility
(hackernoon.com) -
Transparent Hierarchical Storage Management with Apache Kudu and Impala
(blog.cloudera.com) -
Rendezvous Architecture for Data Science in Production
(towardsdatascience.com)#data-science #software-architecture #DBMS #distributed-systems #big-data
-
How we simplified our Data Ingestion & Transformation Process
(engineering.grab.com)#data-pipeline #software-architecture #distributed-systems #backend
-
Managing Uber’s Data Workflows at Scale
(eng.uber.com)#data-pipeline #DBMS #scaling #distributed-systems #big-data
-
Rendezvous with Kafka : A simple guide to get started
(medium.com) -
Presentation: The Whys and Hows of Database Streaming
(www.infoq.com)#data-pipeline #stream-processing #DBMS #distributed-systems
-
Presentation: Massively Scaling MySQL using Vitess
(www.infoq.com) -
Presentation: Patterns of Streaming Applications
(www.infoq.com)#data-pipeline #stream-processing #software-architecture #distributed-systems
-
ValuStor — a memcached alternative built on Scylla
(www.scylladb.com) -
Divide and conquer: Scale your Node.js app using distributed queues
(blog.logrocket.com) -
Myntra Scheduler Service
(medium.com) -
Consistent Hashing Simplified
(towardsdatascience.com) -
Presentation: Point-to-Point Messaging Architecture - The Reactive Endgame
(www.infoq.com) -
Presentation: Cloud-native Streaming Platform: Running Apache Kafka on PKS
(www.infoq.com) -
A Beginner’s Perspective on Kafka Streams: Building Real-Time Walkthrough Detection
(www.confluent.io)#data-pipeline #stream-processing #distributed-systems #apache-kafka
-
Bridging Offline and Nearline Computations with Apache Calcite
(engineering.linkedin.com)#data-pipeline #software-architecture #distributed-systems #backend
-
Managing Distributed Tasks with Helix Task Framework
(engineering.linkedin.com)#software-architecture #distributed-systems #backend #task-queue
-
CockroachDB's Consistency Model
(www.cockroachlabs.com) -
Big Data Metrics Discovery
(engineering.salesforce.com)#software-architecture #distributed-systems #big-data #backend
-
Application Scaling with Raft
(rockthecode.io) -
Migrating Kafka's Zookeeper With No Downtime
(engineeringblog.yelp.com) -
Presentation: Decoding Distributed Systems
(www.infoq.com) -
Pipelining Consensus Writes to Speed Up Distributed SQL Transactions
(www.cockroachlabs.com) -
Introducing ThirdEye: LinkedIn’s Business-Wide Monitoring Platform
(engineering.linkedin.com)#software-design #software-architecture #scaling #distributed-systems #backend
-
Designing resilient systems: Circuit Breakers or Retries? (Part 2)
(engineering.grab.com)#software-architecture #infra #distributed-systems #availability
-
Exploring Distributed System Theory: Availability and Consistency
(hackernoon.com) -
Lessons I’ve Learned Building Distributed Systems with CQRS and Event Sourcing
(hackernoon.com)#software-architecture #distributed-systems #backend #DDD #CQRS
-
Cape Technical Deep Dive
(blogs.dropbox.com)#software-architecture #distributed-systems #event-queue #backend
-
Running Apache Airflow At Lyft
(eng.lyft.com)#data-pipeline #software-architecture #distributed-systems #backend
-
Native secondary indexing in Manhattan
(blog.twitter.com)#software-architecture #DBMS #distributed-systems #systems #data-modeling
-
Designing resilient systems: Circuit Breakers or Retries? (Part 1)
(engineering.grab.com)#software-design #software-architecture #distributed-systems #backend
-
From Code Quality to Integration: Optimizing Alibaba’s Blink Testing Framework
(hackernoon.com)#software-architecture #QA #testing #optimisation #distributed-systems
-
Presentation: CRDTs in Production
(www.infoq.com)#software-architecture #DBMS #scaling #distributed-systems #systems
-
Envoy Proxy at Reddit
(redditblog.com) -
Achieving high availability with stateful Kafka Streams applications
(tech.transferwise.com) -
How to build a fault tolerant system?
(www.future-processing.pl)#software-architecture #algorithms #distributed-systems #decentralised
-
Cross-Shard Queries & Lookup Tables
(medium.com) -
Shard Splits with Consistent Snapshots
(medium.com) -
Reasons to Scale Horizontally
(blog.wallaroolabs.com) -
Abstracting Sharding with Vitess and Distributed Deadlocks
(medium.com) -
Splitting Stateful Services across Continents at Instagram
(www.infoq.com) -
Airflow Architecture at Drivy
(drivy.engineering)#software-architecture #distributed-systems #orchestration #task-queue
-
What’s the big deal about Decentralized Consensus?
(blog.insightdatascience.com)#algorithms #distributed-systems #block-chain #decentralised
-
MySQL High Availability On-Premises: A Geographically Distributed Scenario
(www.percona.com) -
Kafka Connect Deep Dive – Converters and Serialization Explained
(www.confluent.io)#data-pipeline #distributed-systems #apache-kafka #internals #backend
-
Federated learning: distributed machine learning with data locality and privacy
(blog.fastforwardlabs.com)#software-architecture #machine-learning #distributed-systems #privacy
-
Cross shard transactions at 10 million requests per second
(blogs.dropbox.com)#software-design #software-architecture #scaling #distributed-systems #backend
-
Running Apache Kafka on Kubernetes at Shopify
(shopifyengineering.myshopify.com) -
Blockchains vs. Hashgraphs
(hackernoon.com) -
The Treacherous Tangle of Redundant Data: Resilience for Wallaroo
(blog.wallaroolabs.com)#distributed-systems #data-stream #fault-tolerance #systems #reliability
-
Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads
(eng.uber.com)#data-pipeline #software-architecture #distributed-systems #big-data #backend
-
Apache Kafka’s Distributed System Firefighter — The Controller Broker
(hackernoon.com) -
Scaling up with Distributed Tensorflow on Spark
(towardsdatascience.com) -
Anomaly Detection in Zipkin Trace Data
(engineering.salesforce.com) -
Presentation: Modeling the Real World with Elixir/OTP
(www.infoq.com) -
Causal guarantees are anything but casual
(engineering.mongodb.com) -
Building the Contacts Platform at LinkedIn
(engineering.linkedin.com)#data-pipeline #software-architecture #distributed-systems #backend
-
Open Sourcing Subzero
(medium.com) -
Presentation: Help! I Accidentally Distributed My System!
(www.infoq.com)#software-engineering #software-architecture #distributed-systems
-
Dweb: Decentralised, Real-Time, Interoperable Communication with Matrix
(hacks.mozilla.org) -
Uber’s Big Data Platform: 100+ Petabytes with Minute Latency
(eng.uber.com)#software-architecture #distributed-systems #big-data #systems
-
The Netflix Media Database (NMDB)
(medium.com) -
Lessons learned from creating a rich-text editor with real-time collaboration
(medium.com)#algorithms #distributed-systems #real-time #frontend #editor
-
ZooKeeper at Twitter
(blog.twitter.com) -
Checkpointing and Consistent Recovery Lines: How We Handle Failure in Wallaroo
(blog.wallaroolabs.com)#software-architecture #algorithms #distributed-systems #fault-tolerance
-
Managing data store locality at scale with Akkio
(code.fb.com) -
Open Sourcing Mirus
(engineering.salesforce.com)#data-pipeline #software-architecture #distributed-systems #apache-kafka
-
A Brief History of High Availability
(www.cockroachlabs.com) -
Understanding Lamport Timestamps with Python’s multiprocessing library
(towardsdatascience.com) -
Presentation: Real-time, Fine-grained Version Control with CRDTs
(www.infoq.com)#algorithms #distributed-systems #data-structures #real-time #synchronisation
-
Big Data Governance: Hive Metastore Listener for Apache Atlas Use Cases
(www.ebayinc.com) -
Presentation: Complex Event Flows in Distributed Systems
(www.infoq.com)#software-architecture #distributed-systems #backend #fault-tolerance #state-machine
-
Towards a distributed and real-time framework for robots
(hackernoon.com) -
Running elasticsearch cluster on spot instances
(medium.com) -
Progress for big data in Kubernetes
(www.oreilly.com) -
Introducing cstar: The Spotify Cassandra orchestration tool, now open source
(labs.spotify.com) -
Alibaba Unveils PolarFS Distributed File System for Cloud Computing
(hackernoon.com) -
Presentation: How Machines Help Humans Root Case Issues @ Netflix
(www.infoq.com) -
Article: A Critique of Resizable Hash Tables: Riak Core & Random Slicing
(www.infoq.com)#software-architecture #algorithms #noSQL #distributed-systems #systems
-
FEDERATED MACHINE LEARNING
(towardsdatascience.com)#software-architecture #machine-learning #distributed-systems
-
The Joys of Circuit Breaking
(medium.com)#software-architecture #scaling #distributed-systems #microservices
-
Scaling at Instacart: Distributing Data Across Multiple Postgres Databases with Rails
(tech.instacart.com) -
Dynamic Keys
(blog.wallaroolabs.com) -
Presentation: Using Chaos to Build Resilient Systems
(www.infoq.com) -
Hidden scaling issues of distributed systems — System design in the real world
(blog.ably.io) -
Herb: Multi-DC Replication Engine for Uber’s Schemaless Datastore
(eng.uber.com) -
Keeping Counts In Sync
(developers.soundcloud.com)#data-pipeline #software-architecture #distributed-systems #apache-kafka
-
Presentation: IoT and Microservices
(www.infoq.com) -
Distributed graphs processing with Spark GraphX
(hackernoon.com) -
How we built a data pipeline with Lambda Architecture using Spark/Spark Streaming
(medium.com)#data-pipeline #distributed-systems #data-stream #apache-spark
-
Presentation: Distributed Tracing: Latency Analysis for Your Microservices Using Spring Cloud & Zipkin
(www.infoq.com) -
Presentation: The Future of Distributed Databases Is Relational
(www.infoq.com) -
Taming the Beast: How Scylla Leverages Control Theory to Keep Compactions Under Control
(www.scylladb.com) -
Build a MapReduce flow in Elixir
(hackernoon.com) -
Article: Picking an Active-Active Geo Distribution Strategy: Comparing Merge Replication and CRDT
(www.infoq.com) -
YARN FairScheduler Preemption Deep Dive
(blog.cloudera.com) -
Connecting Competing Microservices using RabbitMQ
(itnext.io) -
Analytics on Bare Metal: Xenon and Kafka® Connect
(www.confluent.io) -
Solr: Improving performance for Batch Indexing
(blog.box.com) -
Apache Spark - Performance
(blog.scottlogic.com) -
LinkedOut: A Request-Level Failure Injection Framework
(engineering.linkedin.com) -
Building Layr: An Experimental Decentralized Cloud Storage System
(hackernoon.com) -
Concourse: Generating Personalized Content Notifications in Near-Real-Time
(engineering.linkedin.com)#software-architecture #scaling #distributed-systems #backend
-
Improving Response Latency in Elasticsearch with Adaptive Replica Selection
(www.elastic.co) -
Consensus Mechanisms — As Detailed and Concise as possible!
(hackernoon.com)#algorithms #distributed-systems #block-chain #decentralised
-
Distributed ledger consensus explained
(hackernoon.com) -
A Thorough Introduction to Distributed Systems
(hackernoon.com) -
Sleep better at night! Here is how to respawn a MongoDB Server automatically!
(hackernoon.com)#distributed-systems #mongoDB #fault-tolerance #availability
-
Samza Aeon: Latency Insights for Asynchronous One-Way Flows
(engineering.linkedin.com) -
Distributed Cache
(jobs.zalando.com) -
Optimizing CAL Report Hadoop MapReduce Jobs
(www.ebayinc.com) -
@Scale – Part I (Task Queues)
(blog.imaginea.com) -
Consensus Mechanisms Explained: PoW vs. PoS
(hackernoon.com) -
How the end-to-end back-pressure mechanism inside Wallaroo works
(blog.wallaroolabs.com) -
Builders of the Decentralized Web: 10 Of The Most Innovative Technologies
(hackernoon.com) -
Crypto DeepDive: Bloom Protocol
(hackernoon.com) -
A Beginner’s Guide to IPFS
(hackernoon.com) -
Geo-Partitioning: What Global Data Actually Looks Like
(www.cockroachlabs.com) -
Event Sourcing Using Apache Kafka
(www.confluent.io) -
Shattering the Trillion-Rows-Per-Second Barrier With MemSQL
(blog.memsql.com) -
Is BFT Consensus effective for Proof-of-Stake Blockchain Implementations
(hackernoon.com) -
Distributed transactions and why you should care
(towardsdatascience.com) -
Add Some Smarts To Your Change Data Capture
(medium.com) -
Meet Bandaid, the Dropbox service proxy
(blogs.dropbox.com) -
How We Built Wallaroo to Process Millions of Messages/Sec with Microsecond Latencies
(blog.wallaroolabs.com) -
How to build a Kubernetes Horizontal Pod Autoscaler using custom metrics
(sysdig.com) -
Adventures with Memory Barriers and Seastar on Linux
(www.scylladb.com) -
A Hitchhiker’s Guide to Consensus Algorithms
(hackernoon.com) -
Dynamometer: Scale Testing HDFS on Minimal Hardware with Maximum Fidelity
(engineering.linkedin.com) -
Why we wrote our Kafka Client in Pony
(blog.wallaroolabs.com) -
Byzantine Fault Tolerance in Proof-of-stake protocols
(hackernoon.com) -
Monitoring for Distributed and Microservices Deployments
(www.digitalocean.com) -
Anomaly detection with Apache MXNet
(www.oreilly.com)#data-science #machine-learning #distributed-systems #classifier
-
Simplicity by Distributing Complexity
(jobs.zalando.com) -
Measuring Transactional Integrity in Airbnb’s Distributed Payment Ecosystem
(medium.com) -
How We Sped Up Eventually-Consistent Databases Global Syncs Through Caching
(www.pubnub.com) -
Mesos Executor
(allegro.tech) -
Building Conclave: a decentralized, real time, collaborative text editor
(hackernoon.com) -
Transactions in Apache Kafka
(confluentio.wpengine.com) -
A Blockchain Experiment With Apache Kafka
(hackernoon.com) -
Venice Hybrid: Doing Lambda Better
(engineering.linkedin.com) -
Distributed TensorFlow
(www.oreilly.com) -
A distributed computation system for deep learning experiments with Docker Compose and RabbitMQ.
(deezer.io) -
Deep Learning with PyTorch and GPUs on DC/OS
(mesosphere.com) -
Uber Drives Distributed Deep Learning Forward
(mesosphere.com) -
Enabling Exactly-Once in Kafka Streams
(www.confluent.io)#software-architecture #distributed-systems #apache-kafka #task-queue
-
How to bring fast data access to microservice architecture with in-memory data grids
(www.oreilly.com)#software-architecture #performance #distributed-systems #microservices
-
Stateful Multi-Stream Processing in Python with Wallaroo
(blog.wallaroolabs.com)#data-pipeline #stream-processing #distributed-systems #python
-
Couchbase Ecosystem at LinkedIn
(engineering.linkedin.com)#software-architecture #scaling #distributed-systems #couchbase
-
Using “Follow-the-Workload” to Beat the Latency-Survivability Tradeoff in CockroachDB
(www.cockroachlabs.com)#software-architecture #performance #distributed-systems #cockroachDB
-
Running Kafka Streams applications in AWS
(jobs.zalando.com)#data-pipeline #stream-processing #distributed-systems #apache-kafka
-
Real-time Ranking with Apache Kafka’s Streams API
(jobs.zalando.com)#stream-processing #distributed-systems #apache-kafka #pubsub
-
Transactions in Apache Kafka
(www.confluent.io) -
Scaling Distributed Joins
(blog.memsql.com) -
How Hinted Handoff Works in Scylla
(www.scylladb.com) -
Reliability under abnormal conditions — Part One
(www.thoughtworks.com) -
Graceful Degradation in Spark Streaming Applications
(eng.wealthfront.com) -
Scalable, Distributed Secondary Indexing in Scylla
(www.scylladb.com) -
High Availability: Concepts and Theory
(hackernoon.com) -
Microservices are hard — an invaluable guide to microservices.
(hackernoon.com) -
How Shopify Governs Containers at Scale with Grafeas and Kritis
(shopifyengineering.myshopify.com) -
Containers and Distributed Systems: Where They Came From and Where They’re Going
(mesosphere.com) -
The NEW API Pattern
(developers.redhat.com)#software-architecture #API #distributed-systems #RPC #history
-
Scalable SQL Made Easy: How CockroachDB Automates Operations
(www.cockroachlabs.com) -
Ripple: The Most (Demonstrably) Scalable Blockchain
(highscalability.com) -
Analyzing distributed trace data
(medium.com) -
Low-risk Monolith to Microservice Evolution Part I
(developers.redhat.com) -
Disaster Recovery for Multi-Datacenter Apache Kafka Deployments
(www.confluent.io) -
How does it work? Docker! Part 3: Load balancing, service discovery and security!
(blog.octo.com) -
Dissect Helps Engineers Visualize and Debug Distributed Applications
(www.ebaytechblog.com) -
Crossing the Streams – Joins in Apache Kafka
(www.confluent.io) -
Scaling Event Sourcing for Netflix Downloads, Episode 2
(medium.com) -
It’s Okay To Store Data In Apache Kafka
(www.confluent.io) -
How Apache Kafka is Tested
(www.confluent.io) -
Cells in the Cloud: Distributed Runtime Prototype Implementation
(software.intel.com) -
How We Optimize Apache Spark Jobs
(rea.tech) -
Elasticsearch: Adventures in scaling a multitenant platform
(www.bigeng.io) -
Publishing with Apache Kafka at The New York Times
(www.confluent.io) -
Using the Data Access Object Pattern in a Microservice Architecture
(blog.codeship.com) -
The Total Newbie’s Guide to Cassandra
(blog.insightdatascience.com) -
Stream Processing with Apache Flink and DC/OS
(mesosphere.com) -
LogDevice: a distributed data store for logs
(code.facebook.com) -
Building a Distributed Work Queue with MySQL and RabbitMQ
(eng.wealthfront.com) -
Open Sourcing Kafka Cruise Control
(engineering.linkedin.com) -
Tessa: 1,000,000,000 Strava Activities, 1 Spatiotemporal Dataset
(medium.com) -
Open sourcing DoctorKafka: Kafka cluster healing and workload balancing
(medium.com) -
Understanding Failure Modes in Message and Event-based Systems
(multithreaded.stitchfix.com) -
5 Technologies We Have Used At Hootsuite to Build a Flexible Distributed Data PipeLine
(code.hootsuite.com)#dev-tools #software-architecture #infra #distributed-systems #backend
-
Leveraging the Power of a Database ‘Unbundled’
(www.confluent.io) -
Kubernetes at GitHub
(githubengineering.com) -
The BlackRock Messaging System
(rockthecode.io) -
The Simplest Useful Kafka Connect Data Pipeline In The World … or Thereabouts (Part 1)
(www.confluent.io) -
Organizing Microservices – Modern Integration
(developers.redhat.com) -
Microservices and Docker containers: Architecture, Patterns and Development guidance
(blogs.msdn.microsoft.com) -
Messaging as the Single Source of Truth
(www.confluent.io) -
Migrating From Oracle to Espresso
(engineering.linkedin.com) -
Powering Helix’s Auto Rebalancer with Topology-Aware Partition Placement
(engineering.linkedin.com)#scaling #distributed-systems #cluster-management #fault-tolerance
-
Where Do I Put ProxySQL?
(www.percona.com) -
Moving Yelp's Core Business Search to Elasticsearch
(engineeringblog.yelp.com)#software-architecture #scaling #distributed-systems #elastic
-
Hacking Blockchain
(capgemini.github.io) -
Presto - a small step for DevOps engineer but a big step for BigData analyst
(allegro.tech)