Tagged | data-engineering

• Designing Tinder

  (highscalability.com)

  #software-design #software-architecture #infra #data-engineering

• Auto-Diagnosis and Remediation in Netflix Data Platform

  (netflixtechblog.com)

  #data-pipeline #infra #data-engineering

• Improving Reliability: Building a Vitess Balancer to Minimize MySQL Downtime

  (product.hubspot.com)

  #DBMS #scaling #distributed-systems #data-engineering

• Pinterest Druid Holiday Load Testing

  (stackshare.io)

  #DBMS #performance #scaling #data-engineering

• Presentation: Robust Foundation for Data Pipelines at Scale - Lessons from Netflix

  (www.infoq.com)

  #data-pipeline #performance #scaling #data-engineering

• Evolving LinkedIn’s analytics tech stack

  (engineering.linkedin.com)

  #data-pipeline #infra #analytics #data-engineering

• eBay’s Global Secondary Indexes

  (tech.ebayinc.com)

  #scaling #distributed-systems #data-engineering

• MemQ: An efficient, scalable cloud native PubSub system

  (medium.com)

  #software-architecture #scaling #distributed-systems #data-engineering

• How Uber Migrated Financial Data from DynamoDB to Docstore

  (eng.uber.com)

  #software-engineering #software-architecture #DBMS #migration #data-engineering

• Scaling Apache Druid for Real-Time Cloud Analytics at Confluent

  (www.confluent.io)

  #scaling #distributed-systems #cloud #data-engineering

• 4 Key Design Principles and Guarantees of Streaming Databases

  (www.confluent.io)

  #data-pipeline #software-architecture #DBMS #data-engineering

• CarbonJ: A high performance, high-scale, drop-in replacement for carbon-cache and carbon-relay

  (engineering.salesforce.com)

  #software-architecture #scaling #distributed-systems #data-engineering

• How we built a forever-free serverless SQL database

  (www.cockroachlabs.com)

  #software-architecture #DBMS #scaling #data-engineering

• Introducing uGroup: Uber’s Consumer Management Framework

  (eng.uber.com)

  #software-architecture #distributed-systems #data-engineering

• Processing billions of events in real time at Twitter

  (blog.twitter.com)

  #software-architecture #scaling #distributed-systems #data-engineering

• How to ETL at Petabyte-Scale with Trino

  (engineering.salesforce.com)

  #scaling #big-data #data-engineering

• Improving HDFS I/O Utilization for Efficiency

  (eng.uber.com)

  #performance #distributed-systems #big-data #data-engineering

• Scaling indexing and search - Algolia New Search Architecture Part 2

  (highscalability.com)

  #DBMS #search #systems #data-engineering

• An Engineer's Guide to Building a Database for Data-Intensive Applications

  (www.singlestore.com)

  #DBMS #systems #data-engineering

• Evolution of Region Assignment in the Apache HBase Architecture — Part 3

  (engineering.salesforce.com)

  #performance #scaling #distributed-systems #data-engineering

• Search indexing optimisation

  (engineering.grab.com)

  #DBMS #search #data-engineering

• Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot

  (eng.uber.com)

  #software-architecture #distributed-systems #data-engineering

• Migrating to Elasticsearch with dense vector for Carousell Spotlight search engine

  (medium.com)

  #DBMS #search #performance #data-engineering

• Scaling LinkedIn's Hadoop YARN cluster beyond 10,000 nodes

  (engineering.linkedin.com)

  #scaling #distributed-systems #data-engineering

• Jellyfish: Cost-Effective Data Tiering for Uber’s Largest Storage System

  (eng.uber.com)

  #software-architecture #performance #distributed-systems #data-engineering

• Pinterest’s Analytics as a Platform on Druid (Part 3 of 3)

  (medium.com)

  #DBMS #distributed-systems #analytics #data-engineering

• Cost-Efficient Open Source Big Data Platform at Uber

  (eng.uber.com)

  #optimisation #distributed-systems #big-data #data-engineering

• Challenges and Opportunities to Dramatically Reduce the Cost of Uber’s Big Data

  (eng.uber.com)

  #software-engineering #big-data #data-engineering

• How we built a general purpose key value store for Facebook with ZippyDB

  (engineering.fb.com)

  #DBMS #distributed-systems #systems #data-engineering

• How we scaled the size of Pinterest’s ad corpus by 60x

  (medium.com)

  #performance #scaling #systems #data-engineering

• How Airbnb Built “Wall” to prevent data bugs

  (medium.com)

  #software-design #software-architecture #QA #data-engineering

• The Antidote for Data Architecture Complexity: A Unified Database

  (www.singlestore.com)

  #DBMS #performance #data-engineering

• How to Identify and Tune a Problematic Query with SQL EXPLAIN

  (www.cockroachlabs.com)

  #debugging #DBMS #SQL #data-engineering

• ‘Orders Near You’ and User-Facing Analytics on Real-Time Geospatial Data

  (eng.uber.com)

  #software-architecture #big-data #GeoData #data-engineering

• Building scalable near-real time indexing on HBase

  (medium.com)

  #software-architecture #scaling #data-engineering

• From daily dashboards to enterprise grade data pipelines

  (engineering.linkedin.com)

  #data-pipeline #software-architecture #data-engineering

• Unified Flink Source at Pinterest: Streaming Data Processing

  (medium.com)

  #data-pipeline #software-architecture #data-engineering

• Interactive Querying with Apache Spark SQL at Pinterest

  (medium.com)

  #DBMS #scaling #big-data #data-engineering

• Improving data processing efficiency using partial deserialization of Thrift

  (medium.com)

  #performance #scaling #backend #data-engineering

• Article: Building Latency Sensitive User Facing Analytics via Apache Pinot

  (www.infoq.com)

  #distributed-systems #analytics #real-time #data-engineering

• Scaling Worldwide Parcel Logistics with SingleStore and Vectorized

  (www.singlestore.com)

  #DBMS #performance #scaling #data-engineering

• Grab App at Scale with Scylla

  (www.scylladb.com)

  #software-architecture #scaling #backend #data-engineering

• Consolidating Facebook storage infrastructure with Tectonic file system

  (engineering.fb.com)

  #software-architecture #big-data #systems #data-engineering

• Giving the power of data in hands of your data analyst

  (lambda.grofers.com)

  #software-architecture #analytics #data-engineering

• Realtime and databases — a discussion on coupling versus modularity

  (ably.com)

  #DBMS #scaling #real-time #data-engineering

• Optimisation using Sparklens

  (medium.com)

  #performance #apache-spark #data-engineering

• Building a Version-Controlled Data Aquarium

  (benchling.engineering)

  #data-science #software-engineering #data-engineering

• Sharding, simplification, and Twitter’s ads serving platform

  (blog.twitter.com)

  #software-architecture #performance #data-engineering

• How We Built REGIONAL BY ROW for Row-Level Data Homing & Distribution

  (www.cockroachlabs.com)

  #DBMS #distributed-systems #data-engineering

• Optimizing Analytics Data Processing on eBay’s New Open-Source-Based Platform

  (tech.ebayinc.com)

  #data-pipeline #analytics #big-data #data-engineering

• What actually is a Data Mesh? And is it really a thing?

  (blog.scottlogic.com)

  #data-pipeline #infra #data-engineering

• Integrated BlobDB

  (rocksdb.org)

  #DBMS #systems #data-engineering

• The exabyte club: LinkedIn’s journey of scaling the Hadoop Distributed File System

  (engineering.linkedin.com)

  #scaling #distributed-systems #analytics #big-data #data-engineering

• Building a Label-Based Enforcement Pipeline for Trust & Safety

  (medium.com)

  #data-pipeline #software-architecture #data-engineering

• Vinted Search Scaling Chapter 4: Query Log

  (engineering.vinted.com)

  #data-pipeline #search #scaling #data-engineering

• From Vendor to In-house: How eBay Reimagined Its Analytics Landscape

  (tech.ebayinc.com)

  #data-pipeline #analytics #data-engineering

• Shallow Mirror

  (medium.com)

  #performance #apache-kafka #data-engineering

• Presentation: Change Data Capture for Distributed Databases @Netflix

  (www.infoq.com)

  #infra #DBMS #distributed-systems #data-engineering

• Let me automate that for you II, Electric Bugaloo

  (tech.gc.com)

  #software-engineering #automation #performance #data-engineering

• How we made DISTINCT queries up to 8000x faster on PostgreSQL

  (blog.timescale.com)

  #DBMS #performance #systems #data-engineering

• How Airbnb Achieved Metric Consistency at Scale

  (medium.com)

  #software-architecture #distributed-systems #analytics #data-engineering

• On Coordinated Omission

  (www.scylladb.com)

  #DBMS #performance #distributed-systems #data-engineering

• Adopting RocksDB within Manhattan

  (blog.twitter.com)

  #scaling #distributed-systems #data-engineering

• Attack of the Delta Clones (Against Disaster Recovery Availability Complexity)

  (databricks.com)

  #software-architecture #infra #distributed-systems #data-engineering

• FullContact: Improving the Graph by Transitioning to Scylla

  (www.scylladb.com)

  #DBMS #distributed-systems #data-engineering

• Solving for the cardinality of set intersection at scale with Pinot and Theta Sketches

  (engineering.linkedin.com)

  #software-architecture #DBMS #scaling #data-engineering

• The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 1

  (engineering.salesforce.com)

  #DBMS #distributed-systems #data-engineering

• Scylla’s New IO Scheduler

  (www.scylladb.com)

  #DBMS #performance #systems #data-engineering

• Powering Messaging Enabledness with Yelp's Data Infrastructure

  (engineeringblog.yelp.com)

  #software-architecture #backend #data-engineering

• Detecting Image Similarity in (Near) Real-time Using Apache Flink

  (medium.com)

  #software-architecture #machine-learning #image-processing #data-engineering

• Pinterest Flink Deployment Framework

  (stackshare.io)

  #backend #cloud #data-engineering

• ReversingLabs: Serving File Reputation for Twenty Billion Files

  (www.scylladb.com)

  #DBMS #scaling #data-engineering

• One billion files in Ozone

  (blog.cloudera.com)

  #DBMS #performance #distributed-systems #data-engineering

• Presentation: Scalable, Cloud-native Data Applications by Example

  (www.infoq.com)

  #scaling #distributed-systems #cloud #data-engineering

• iFood Relies on Scylla to Deliver Over 100 Million Events a Month to Restaurants

  (www.scylladb.com)

  #DBMS #noSQL #scaling #data-engineering

• Learning Multi-dimensional indices: The next big thing in OLAP DBs

  (towardsdatascience.com)

  #DBMS #big-data #data-engineering

• Powering Pinterest Ads Analytics with Apache Druid

  (stackshare.io)

  #noSQL #analytics #druid #data-engineering

• Real-Time Data Replication with ksqlDB

  (www.confluent.io)

  #DBMS #apache-kafka #data-engineering

• How we scaled Graphite to 100,000 writes per second.

  (medium.com)

  #DBMS #performance #scaling #data-engineering

• Presentation: Streaming a Million likes/second: Real-time Interactions on Live Video

  (www.infoq.com)

  #software-architecture #scaling #distributed-systems #data-engineering

• Enabling HDFS Federation Having 1B File System Objects

  (tech.ebayinc.com)

  #DBMS #scaling #distributed-systems #data-engineering

• Getting storage engines ready for fast storage devices

  (engineering.mongodb.com)

  #software-architecture #DBMS #systems #data-engineering

• Hash Sharded Indexes Unlock Linear Scaling for Sequential Workloads

  (www.cockroachlabs.com)

  #DBMS #distributed-systems #data-engineering

• Augury: Insights into Industrial IoT Time-Series Data

  (www.scylladb.com)

  #DBMS #IoT #time-series #data-engineering

• Using Kafka to Throttle QPS on MySQL Shards in Bulk Write APIs

  (stackshare.io)

  #software-architecture #DBMS #scaling #apache-kafka #data-engineering

• How we improved latency through projection in Espresso

  (engineering.linkedin.com)

  #software-architecture #DBMS #distributed-systems #data-engineering

• Architecture for High-Throughput Low-Latency Big Data Pipeline on Cloud

  (towardsdatascience.com)

  #DBMS #big-data #cloud #data-engineering

• Bucketisation: Using cassandra for time series data scans.

  (medium.com)

  #DBMS #analytics #time-series #data-engineering

• How Netflix uses Druid for Real-time Insights to Ensure a High-Quality Experience

  (netflixtechblog.com)

  #DBMS #distributed-systems #analytics #real-time #data-engineering

• Presentation: Data Mesh Paradigm Shift in Data Platform Architecture

  (www.infoq.com)

  #infra #devops #data-engineering

• Nauto: Achieving Consistency in an Eventually Consistent Environment

  (www.scylladb.com)

  #DBMS #algorithms #distributed-systems #data-engineering

• Integrating Elasticsearch and ksqlDB for Powerful Data Enrichment and Analytics

  (www.confluent.io)

  #DBMS #apache-kafka #analytics #data-engineering

• How to enable data scientists to stop managing ETL pipelines and get back to doing data science: Part I

  (tech.wayfair.com)

  #data-pipeline #data-science #infra #data-engineering

• Building a Materialized Cache with ksqlDB

  (www.confluent.io)

  #DBMS #apache-kafka #data-engineering

• FireEye: Providing Real-Time Threat Analysis using a Graph Database

  (www.scylladb.com)

  #DBMS #analytics #real-time #graph-processing #data-engineering

• Presentation: Snowflake Architecture: Building a Data Warehouse for the Cloud

  (www.infoq.com)

  #DBMS #distributed-systems #cloud #data-engineering

• Spotify Unwrapped: How we brought you a decade of data

  (labs.spotify.com)

  #analytics #big-data #data-engineering

• Data Migrations Don’t Have to Come with Downtime

  (engblog.nextdoor.com)

  #software-engineering #migration #backend #data-engineering

• Fanatics: Using Scylla for Online Order Capture

  (www.scylladb.com)

  #DBMS #scaling #big-data #data-engineering

• Infinite Storage in Confluent Platform

  (www.confluent.io)

  #distributed-systems #apache-kafka #big-data #data-engineering

• Streams and Monk – How Yelp is Approaching Kafka in 2020

  (engineeringblog.yelp.com)

  #data-pipeline #apache-kafka #backend #data-engineering

• Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing

  (eng.uber.com)

  #data-pipeline #software-architecture #distributed-systems #data-engineering

• Speeding Up SELECT Queries with Parquet Page Indexes

  (blog.cloudera.com)

  #DBMS #performance #SQL #data-engineering

• Stop the Insanity: Eliminating Data Infrastructure Sprawl

  (www.memsql.com)

  #infra #DBMS #distributed-systems #data-engineering

• Maximizing Disk Utilization with Incremental Compaction

  (www.scylladb.com)

  #software-architecture #DBMS #compression #data-engineering

• Streams and Tables in Apache Kafka: Elasticity, Fault Tolerance, and Other Advanced Concepts

  (www.confluent.io)

  #data-pipeline #DBMS #distributed-systems #data-engineering

• Engineering SQL Support on Apache Pinot at Uber

  (eng.uber.com)

  #DBMS #distributed-systems #SQL #big-data #data-engineering

• Reliably Upgrading Apache Airflow at Slack’s Scale

  (slack.engineering)

  #distributed-systems #migration #backend #data-engineering

• Comcast: Sprinting from Cassandra to Scylla

  (www.scylladb.com)

  #software-architecture #performance #distributed-systems #data-engineering

• Streams and Tables in Apache Kafka: Topics, Partitions, and Storage Fundamentals

  (www.confluent.io)

  #DBMS #distributed-systems #apache-kafka #data-engineering

• Streams and Tables in Apache Kafka: A Primer

  (www.confluent.io)

  #data-pipeline #DBMS #apache-kafka #data-engineering

• Introducing Flyte: Cloud Native Machine Learning and Data Processing Platform

  (eng.lyft.com)

  #data-pipeline #machine-learning #cloud #data-engineering

• Presentation: Practical Change Data Streaming Use Cases with Apache Kafka & Debezium

  (www.infoq.com)

  #scaling #apache-kafka #kubernetes #data-engineering

• How I'm Engineering a Versioned Database Storage Engine for Byte-Addressable NVM

  (hackernoon.com)

  #DBMS #algorithms #performance #systems #data-engineering

• Presentation: Scaling Beyond a Billion Transactions Per Day with Sub-second Responses

  (www.infoq.com)

  #software-architecture #infra #performance #scaling #data-engineering

• How ads indexing works at Pinterest

  (medium.com)

  #data-pipeline #software-architecture #scaling #data-engineering

• Streaming Cassandra into Kafka in (Near) Real-Time: Part 2

  (engineeringblog.yelp.com)

  #data-pipeline #distributed-systems #real-time #data-engineering

• The Story Behind MemSQL’s Skiplist Indexes

  (www.memsql.com)

  #DBMS #algorithms #systems #data-engineering

• DBLog: A Generic Change-Data-Capture Framework

  (medium.com)

  #software-architecture #DBMS #scaling #data-engineering

• Uber’s Data Platform in 2019: Transforming Information to Intelligence

  (eng.uber.com)

  #data-pipeline #scaling #distributed-systems #data-engineering

• GokuL: Extending time series data storage to serve beyond one day

  (medium.com)

  #DBMS #scaling #time-series #data-engineering

• How Scylla Scaled to One Billion Rows a Second

  (www.scylladb.com)

  #scaling #distributed-systems #big-data #data-engineering

• Podcast: Josh Wills on Building Resilient Data Engineering and Machine Learning Products at Slack

  (www.infoq.com)

  #software-architecture #machine-learning #scaling #data-engineering

• Presentation: Batch Processing in 2019

  (www.infoq.com)

  #data-pipeline #software-architecture #backend #data-engineering

• Streaming Cassandra into Kafka in (Near) Real-Time: Part 1

  (engineeringblog.yelp.com)

  #data-pipeline #software-architecture #distributed-systems #data-engineering

• Presentation: Future of Data Engineering

  (www.infoq.com)

  #data-pipeline #DBMS #scaling #data-engineering

• Reducing Multi-Region Latency with Follower Reads

  (www.cockroachlabs.com)

  #DBMS #performance #distributed-systems #data-engineering

• Availability and Region Failure: Joint Consensus in CockroachDB

  (www.cockroachlabs.com)

  #DBMS #scaling #distributed-systems #reliability #data-engineering

• Using Kafka to throttle QPS on MySQL shards in bulk write APIs

  (medium.com)

  #DBMS #apache-kafka #SQL #data-engineering

• A Glimpse into the World of Embedded Database Feat. RocksDB

  (medium.com)

  #DBMS #internals #systems #data-engineering

• Maximizing Performance via Concurrency While Minimizing Timeouts in Distributed Databases

  (www.scylladb.com)

  #DBMS #performance #distributed-systems #concurrency #data-engineering

• Unpacking Competitive Benchmark Claims

  (www.cockroachlabs.com)

  #DBMS #performance #data-engineering

• Spotify’s Event Delivery – Life in the Cloud

  (labs.spotify.com)

  #software-architecture #big-data #data-engineering

• Parallel Commits: An Atomic Commit Protocol For Globally Distributed Transactions

  (www.cockroachlabs.com)

  #DBMS #distributed-systems #data-engineering

• Optimizing Search Index Generation using secondary cache

  (medium.com)

  #performance #distributed-systems #big-data #caching #data-engineering

• Building columnar compression in a row-oriented database

  (blog.timescale.com)

  #DBMS #time-series #compression #data-engineering

• How We Built a Vectorized SQL Engine

  (www.cockroachlabs.com)

  #software-architecture #DBMS #SQL #data-engineering

• Open Sourcing Amundsen: A Data Discovery And Metadata Platform

  (eng.lyft.com)

  #software-architecture #backend #data-engineering

• An inside look at LinkedIn’s data pipeline monitoring system

  (engineering.linkedin.com)

  #data-pipeline #software-architecture #monitoring #data-engineering

• Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store

  (towardsdatascience.com)

  #machine-learning #big-data #data-engineering

• 2019 @Scale Conference recap

  (engineering.fb.com)

  #software-architecture #scaling #backend #data-engineering

• ML Platform Meetup: Infra for Contextual Bandits and Reinforcement Learning

  (medium.com)

  #data-science #machine-learning #data-engineering

• Evolving Michelangelo Model Representation for Flexibility at Scale

  (eng.uber.com)

  #data-science #machine-learning #scaling #data-engineering

• Delta: A Data Synchronization and Enrichment Platform

  (medium.com)

  #software-architecture #algorithms #distributed-systems #data-engineering

• The Beauty of a Shared-Nothing SQL DBMS for Skewed Database Sizes

  (www.memsql.com)

  #DBMS #scaling #data-engineering

• Compression in Scylla, Part Two

  (www.scylladb.com)

  #DBMS #algorithms #compression #data-engineering

• How LinkedIn customizes Apache Kafka for 7 trillion messages per day

  (engineering.linkedin.com)

  #performance #scaling #distributed-systems #apache-kafka #data-engineering

• Compression in Scylla, Part One

  (www.scylladb.com)

  #DBMS #algorithms #compression #data-engineering

• What Makes Apache Flink Scale?

  (medium.com)

  #analytics #big-data #systems #data-engineering

• How Shopify Manages Petabyte Scale MySQL Backup and Restore

  (engineering.shopify.com)

  #DBMS #scaling #big-data #data-engineering

• Adaptive Throttling of Indexing for Improved Query Responsiveness

  (medium.com)

  #DBMS #performance #scaling #data-engineering

• Multiplexing (Mux) in ProxySQL: Use Case

  (www.percona.com)

  #DBMS #performance #SQL #data-engineering

• PinalyticsDB: A Time Series Database on top of Hbase

  (medium.com)

  #DBMS #analytics #time-series #data-engineering

• Shared Transactional Tables: The Foundation of Next Generation Big Data Warehousing

  (blog.cloudera.com)

  #DBMS #distributed-systems #big-data #data-engineering

• Presentation: Datadog: A Real-time Metrics Database for One Quadrillion Points/Day

  (www.infoq.com)

  #DBMS #analytics #real-time #data-engineering

• Joining Petabytes of Data Per Day: How LiveRamp Powers its Matching Product

  (liveramp.com)

  #DBMS #performance #scaling #data-engineering

• Presentation: Scaling DB Access for Billions of Queries per Day @PayPal

  (www.infoq.com)

  #software-architecture #DBMS #GoLang #data-engineering

• Presentation: CockroachDB: Architecture of a Geo-distributed SQL Database

  (www.infoq.com)

  #software-architecture #DBMS #algorithms #distributed-systems #data-engineering

• Time-Based Anti-Patterns for Caching Time-Series Data

  (www.scylladb.com)

  #DBMS #performance #time-series #data-engineering

• A Technical Introduction to MemSQL

  (www.memsql.com)

  #software-architecture #DBMS #distributed-systems #data-engineering

• Cultivating your Data Lake

  (stackshare.io)

  #data-pipeline #software-architecture #infra #data-engineering

• Replicating PostgreSQL into MemSQL’s Columnstore

  (www.memsql.com)

  #DBMS #performance #PostgreSQL #data-engineering

• How to manage your Snowflake spend with Periscope and dbt

  (about.gitlab.com)

  #infra #performance #scaling #data-engineering

• Presto Infrastructure at Lyft

  (eng.lyft.com)

  #infra #scaling #distributed-systems #backend #data-engineering

• Building a distributed time-series database on PostgreSQL

  (blog.timescale.com)

  #software-architecture #DBMS #time-series #PostgreSQL #data-engineering

• Adventures in big data wonderland: Going down the Pinterest Path

  (medium.com)

  #software-architecture #DBMS #big-data #data-engineering

• Data Hub: A Generalized Metadata Search & Discovery Tool

  (engineering.linkedin.com)

  #software-architecture #search #big-data #data-engineering

• Presentation: Tackling Computing Challenges @CERN

  (www.infoq.com)

  #performance #big-data #computation #data-engineering

• Data Engineering in Badoo: Handling 20 Billion Events per Day

  (www.infoq.com)

  #data-pipeline #software-architecture #scaling #data-engineering

• Data first, SLA always

  (engineering.grab.com)

  #software-design #software-architecture #backend #data-engineering

• Advances in Spam Detection on Tumblr

  (engineering.tumblr.com)

  #data-science #machine-learning #data-engineering

• Extending Hive Replication: Transactional Tables, External Tables, and Statistics

  (blog.cloudera.com)

  #software-architecture #DBMS #data-engineering

• OrderedAppend: An optimization for range partitioning

  (blog.timescale.com)

  #DBMS #algorithms #optimisation #data-engineering

• Maptype — fast doc-value lookups for map data in Elasticsearch

  (engineeringblog.yelp.com)

  #DBMS #algorithms #search #data-engineering

• Improving the scalability of a Spark pipeline for conversion attribution

  (medium.com)

  #data-pipeline #algorithms #performance #data-engineering

• Implementing constraint exclusion for faster query performance

  (timescale.ghost.io)

  #DBMS #performance #data-engineering

• A Scalable SQL Database Powers Real-Time Analytics at Uber

  (www.memsql.com)

  #DBMS #scaling #analytics #data-engineering

• Fast Parallel Testing at Databricks with Bazel

  (databricks.com)

  #QA #testing #distributed-systems #data-engineering

• Accelerating NiFi flows delivery: Part 1

  (blog.octo.com)

  #data-pipeline #software-architecture #performance #optimisation #data-engineering

• Presentation: Automatic Clustering at Snowflake

  (www.infoq.com)

  #infra #DBMS #scaling #distributed-systems #data-engineering

• Petabyte Scale Data Deduplication

  (engineering.mixpanel.com)

  #software-architecture #infra #DBMS #data-engineering

• Making Apache Spark Effortless for All of Uber

  (eng.uber.com)

  #software-architecture #DBMS #distributed-systems #apache-spark #data-engineering

• Open Sourcing Brooklin: Near Real-Time Data Streaming at Scale

  (engineering.linkedin.com)

  #data-pipeline #software-architecture #scaling #distributed-systems #data-engineering

• Auto-Tuning Pinot Real-Time Consumption

  (engineering.linkedin.com)

  #software-architecture #scaling #data-engineering

• Expediting Data Fixes and Data Migrations

  (engineering.linkedin.com)

  #software-engineering #infra #scaling #practices #data-engineering

• OIL+VCache: File abstraction for distributed systems

  (code.fb.com)

  #distributed-systems #systems #data-engineering

• Query Plan Caching in CockroachDB

  (www.cockroachlabs.com)

  #software-architecture #DBMS #performance #data-engineering

• Pilosa: A Scalable High Performance Bitmap Database Index

  (hackernoon.com)

  #DBMS #noSQL #analytics #data-engineering

• Kafka Listeners – Explained

  (www.confluent.io)

  #software-architecture #distributed-systems #apache-kafka #internals #data-engineering

• CockroachDB Change Data Capture: Transactionally and Horizontally Scalable

  (www.cockroachlabs.com)

  #software-architecture #DBMS #internals #data-engineering

• Improving Performance and Capacity for Espresso with New Netty Framework

  (engineering.linkedin.com)

  #software-architecture #performance #distributed-systems #data-engineering

• Community-Focused Feed Optimization

  (engineering.linkedin.com)

  #data-science #software-architecture #machine-learning #analytics #data-engineering

• Building a Scalable Search Architecture

  (www.confluent.io)

  #search #scaling #apache-kafka #data-engineering

• Putting Machine Learning Models into Production

  (blog.cloudera.com)

  #data-science #machine-learning #big-data #production #data-engineering

• Star-Tree Index: Powering Fast Aggregations on Pinot

  (engineering.linkedin.com)

  #DBMS #distributed-systems #internals #data-engineering

• Streaming Data from the Universe with Apache Kafka

  (www.confluent.io)

  #software-architecture #distributed-systems #apache-kafka #data-engineering

• Presentation: Machine Learning Engineering - A New Yet Not so New Paradigm

  (www.infoq.com)

  #software-engineering #machine-learning #practices #data-engineering

• Presentation: Petastorm: A Light-Weight Approach to Building ML Pipelines

  (www.infoq.com)

  #data-pipeline #machine-learning #big-data #data-engineering

• Rethinking the Database Materialized View as an Index

  (blog.timescale.com)

  #DBMS #time-series #big-data #data-engineering

• HDFS Erasure Coding in Production

  (blog.cloudera.com)

  #performance #distributed-systems #hadoop #data-engineering

• Presentation: People You May Know: Fast Recommendations Over Massive Data

  (www.infoq.com)

  #performance #distributed-systems #real-time #graphDB #data-engineering

• Bringing scalable real-time analytics to the enterprise

  (www.oreilly.com)

  #DBMS #scaling #distributed-systems #podcast #data-engineering

• Delos: Simple, flexible storage for the Facebook control plane

  (code.fb.com)

  #software-architecture #DBMS #performance #data-engineering

• Building A Scalable Data Management System for Computer Vision Tasks

  (medium.com)

  #data-pipeline #software-architecture #image-processing #data-engineering

• Log Compacted Topics in Apache Kafka

  (towardsdatascience.com)

  #DBMS #apache-kafka #big-data #data-engineering

• Migrating a Big Data Environment to the Cloud, Part 4

  (liveramp.com)

  #software-architecture #big-data #migration #cloud #data-engineering

• Presentation: Michelangelo Palette: A Feature Engineering Platform at Uber

  (www.infoq.com)

  #data-science #machine-learning #distributed-systems #data-engineering

• Grafana Labs at KubeCon: Awesome Query Performance with Cortex

  (grafana.com)

  #software-architecture #DBMS #noSQL #performance #data-engineering

• Intelligent computing in Snowflake

  (towardsdatascience.com)

  #machine-learning #DBMS #neural-net #data-engineering

• Presentation: Life of a Distributed Graph Database Query

  (www.infoq.com)

  #DBMS #distributed-systems #graph-processing #data-engineering

• Workload Prioritization: Running OLTP and OLAP Traffic on the Same Superhighway

  (www.scylladb.com)

  #DBMS #noSQL #performance #data-engineering

• A Richer Activity, Part 1

  (medium.com)

  #DBMS #SQL #big-data #data-engineering

• Maintainable ETLs: Tips for Making Your Pipelines Easier to Support and Extend

  (multithreaded.stitchfix.com)

  #data-pipeline #software-engineering #practices #data-engineering

• Introducing LINE Games analytics environment

  (engineering.linecorp.com)

  #data-pipeline #software-architecture #big-data #data-engineering

• Accelerating Machine Learning with the Feature Store Service

  (technology.condenast.com)

  #data-science #machine-learning #big-data #data-engineering

• MetricsDB: TimeSeries Database for storing metrics at Twitter

  (blog.twitter.com)

  #software-architecture #DBMS #analytics #time-series #data-engineering

• Continuous aggregates: faster queries with automatically maintained materialized views

  (blog.timescale.com)

  #DBMS #time-series #data-engineering

• Introducing Data Compaction in Ambry

  (engineering.linkedin.com)

  #software-architecture #DBMS #compression #media #data-engineering

• Apache Kafka Data Access Semantics: Consumers and Membership

  (www.confluent.io)

  #distributed-systems #apache-kafka #data-engineering

• Railyard: how we rapidly train machine learning models with Kubernetes

  (stripe.com)

  #data-science #software-engineering #software-architecture #kubernetes #data-engineering

• MySQL InnoDB Sorted Index Builds

  (www.percona.com)

  #DBMS #MySql #internals #data-engineering

• Real-time data processing for monitoring and reporting — A practical use case of spark structured…

  (medium.com)

  #data-pipeline #stream-processing #distributed-systems #apache-spark #data-engineering

• An intuitive understanding of the LAMB optimizer

  (towardsdatascience.com)

  #deep-learning #data-science #performance #data-engineering

• Beam: A Distributed Knowledge Graph Store

  (www.ebayinc.com)

  #DBMS #distributed-systems #GoLang #semantic-data #data-engineering

• Presentation: YugaByte DB - A Planet-scale Database for Low Latency Transactional Apps

  (www.infoq.com)

  #DBMS #performance #scaling #distributed-systems #data-engineering

• Troubleshooting Data Engineering Software

  (engineering.linecorp.com)

  #debugging #performance #distributed-systems #backend #data-engineering

• Consistent Data Partitioning through Global Indexing for Large Apache Hadoop Tables at Uber

  (eng.uber.com)

  #distributed-systems #big-data #hadoop #data-engineering

• Better to Give and to Receive: Alibaba’s Open-source Contributions to Flink

  (hackernoon.com)

  #DBMS #big-data #opensource #data-engineering

• How Bloomberg Tracks Hundreds of Billions of Data Points Daily with MetricTank and Grafana

  (grafana.com)

  #scaling #time-series #big-data #data-engineering

• How eBay Governs its Big Data Fabric

  (www.ebayinc.com)

  #distributed-systems #big-data #hadoop #data-engineering

• Amundsen — Lyft’s data discovery & metadata engine

  (eng.lyft.com)

  #dev-tools #data-science #metadata #data-engineering

• Troubleshooting Data Engineering Software

  (engineering.linecorp.com)

  #debugging #performance #distributed-systems #data-engineering

• Presentation: Michelangelo - Machine Learning @Uber

  (www.infoq.com)

  #data-pipeline #data-science #machine-learning #data-engineering

• Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and…

  (medium.com)

  #software-architecture #infra #scaling #backend #data-engineering

• How to Reindex One Billion Documents in One Hour at SoundCloud

  (developers.soundcloud.com)

  #DBMS #performance #scaling #elastisearch #data-engineering

• Uber Case Study: Choosing the Right HDFS File Format for Your Apache Spark Jobs

  (eng.uber.com)

  #distributed-systems #big-data #hadoop #data-engineering

• Pro Tips: How Booking.com Handles Millions of Metrics Per Second with Graphite

  (grafana.com)

  #performance #scaling #big-data #data-engineering

• Solving Big Data Challenges with Data Science at Uber

  (eng.uber.com)

  #DBMS #scaling #distributed-systems #big-data #data-engineering

• DBEvents: A Standardized Framework for Efficiently Ingesting Data into Uber’s Apache Hadoop Data Lake

  (eng.uber.com)

  #data-pipeline #distributed-systems #hadoop #data-engineering

• Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and More

  (yahooeng.tumblr.com)

  #data-pipeline #software-architecture #backend #data-engineering

• Transparent Hierarchical Storage Management with Apache Kudu and Impala

  (blog.cloudera.com)

  #distributed-systems #big-data #backend #data-engineering