'Speeding Up Uber’s Presto with Alluxio - Part 1' by Alluxiocc: hdfs localcache
At Uber, data informs every decision. Presto is one of the very core engines that powers all sorts of data analytics at Uber. For example, the operations team makes heavy use of Presto for services such as dashboarding; The Uber Eats and marketing teams rely on the results of these queries to make decisions on prices. In addition, Presto is also used in Uber’s compliance department, growth marketing department, and ad-hoc data analytics.
The proxy layer. This layer is responsible for pulling stats from each Presto coordinator, including the number of queries and tasks, CPU and Memory usage, and so on. We determine which cluster each query should be scheduled to based on these stats. In other words, it is doing the load-balancing and query-gating services.
Therefore, we are exploring public cloud services, such as GCS or S3, to serve as an L2 Cache. We want to put some important data or frequently used data sets in the “L2 Cache on the cloud”. For each Presto cluster in the cloud, we plan to leverage local SSDs to cache some data to improve performance. This solution can be scaled into different cloud providers using the same architecture, which is our vision in the long term.
We will discuss the detailed design and improvement of the Alluxio local cache in part 2 of this blog series.The first challenge we encountered was the real time partition updates. At Uber, a lot of tables/partitions are constantly changing because we upsert queries constantly into Hudi tables. As shown below, currently, simple hash mod based node lookup : key 4 % 3 nodes=worker # 1. Now node#3 goes down, new lookup: key 4 % 2 nodes=0, but worker#0 does not have the bytes.Consistent hashing is the solution. Instead of the mod based function, all nodes are placed on a virtual ring. Relative ordering of nodes on the ring doesn’t change, regardless of joining or leaving. Instead of using mode based hash, we always look up the key on the ring.
Österreich Neuesten Nachrichten, Österreich Schlagzeilen
Similar News:Sie können auch ähnliche Nachrichten wie diese lesen, die wir aus anderen Nachrichtenquellen gesammelt haben.
The Economic Consequences of the Peace: Chapter III - The Conference | HackerNoonThe Economic Consequences of the Peace Chapter III: The Conference by John Maynard Keynes is part of HackerNoon’s Book Blog Post series.
Weiterlesen »
This Side of Paradise by F. Scott Fitzgerald - Table of Links | HackerNoon
Weiterlesen »
Parasocial Relationships: A Pervasive Risk | HackerNoonVirtual relationships can be complicated, especially when they're not symmetrical and parasocial: 'I know a lot about you, but you don't know me at all.'
Weiterlesen »
Exploring Mental Health and UX Design with Jennifer Wong | HackerNoonJennifer Wong is the Founder of a UX design boot camp called Product Design Fam and the CEO of Empathie, an app dedicated to empowering racial minorities.
Weiterlesen »
7 Mistakes To Avoid When Hiring An App Development Company | HackerNoonChoosing the right app development company is critical to your project's success. Here are 7 mistakes you should avoid while hiring a development partner.
Weiterlesen »
Interest in Microsoft Wanes 12% While Apple Remains Trendy AF | HackerNoon'Interest in Microsoft Wanes 12% While Apple Remains Trendy AF' apple amazon
Weiterlesen »