Scribe: Transporting petabytes per hour - Engineering at Meta
![Scribe: Transporting petabytes per hour - Engineering at Meta](https://engineering.fb.com/wp-content/uploads/2019/10/CB19-046_Code_Blog_Scribe_Post_Hero-1.jpg)
Our hardware infrastructure comprises millions of machines, all of which generate logs that we need to process, store, and serve. The total size of these logs is several petabytes every hour. The o…
![](https://3.bp.blogspot.com/--1AYp6CJP9c/V4E90bu7RDI/AAAAAAAAFwc/vYAqc-Ma90kmuYG-n7N5BspMnK2WcR_aACLcB/s1600/Screen%2BShot%2B2016-07-09%2Bat%2B10.46.06%2BAM.png)
Realtime Data Processing at Facebook
![](https://engineering.fb.com/wp-content/uploads/2019/10/Sys@hero.jpg?w=580&h=326&crop=1)
Scribe: Transporting petabytes per hour - Engineering at Meta
Lohit Vijayarenu - Log Events @ Twitter: Challenges of Handling Billions of Events per Minute, Twitter, database, data
Scribe: Transporting petabytes per hour - Engineering at Meta
![](https://2.bp.blogspot.com/-v64CcJByjvU/V4E901Oa9wI/AAAAAAAAFwg/3EDfw3iJ2VA9PW-CYF9vEBRvhmNberZ1ACKgB/s1600/Screen%2BShot%2B2016-07-09%2Bat%2B10.46.27%2BAM.png)
Realtime Data Processing at Facebook
Yegor Shrayev on LinkedIn: Scribe: Transporting petabytes per hour via a distributed, buffered…
![](https://spectrum.ieee.org/media-library/manufacturing-closeup-photo.jpg?id=25589486&width=824&quality=85)
6 Things to Know About the Biggest Chip Ever Built - IEEE Spectrum
Top 30 Most Influential Papers in The World of Big Data, PDF, No Sql
![](https://engineering.fb.com/wp-content/uploads/2019/10/CB19-046_Code_Blog_Scribe_Post_Hero-1.jpg)
Scribe: Transporting petabytes per hour - Engineering at Meta
![](https://www.researchgate.net/publication/365299104/figure/fig3/AS:11431281096615355@1668185279029/Each-Reader-node-extracts-mini-batches-from-storage-Batches-are-converted-to-tensors_Q320.jpg)
PDF) RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure
![](https://ars.els-cdn.com/content/image/1-s2.0-S1574013721000629-gr3.jpg)
A survey on the Distributed Computing stack - ScienceDirect
![](https://engineering.fb.com/wp-content/uploads/2023/01/Asynchronous-Computation-at-Scale-Hero.png)
Asynchronous computing at Meta: Overview and learnings
![](https://user-images.githubusercontent.com/45159366/190880704-2042e18b-bc60-4b53-8251-2e3628b3083e.png)
GitHub - mikeroyal/Self-Hosting-Guide: Self-Hosting Guide. Learn all about locally hosting (on premises & private web servers) and managing software applications by yourself or your organization. Including Cloud, LLMs, WireGuard, Automation, Home
![](https://i1.rgstatic.net/publication/365299104_RecD_Deduplication_for_End-to-End_Deep_Learning_Recommendation_Model_Training_Infrastructure/links/636ddbe32f4bca7fd04e140e/largepreview.png)
PDF) RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure