Show simple item record

dc.contributor.authorFischer e Silva, Renan
dc.contributor.authorCarpenter, Paul M.
dc.contributor.otherBarcelona Supercomputing Center
dc.date.accessioned2017-10-11T09:20:24Z
dc.date.available2017-10-11T09:20:24Z
dc.date.issued2017-09-26
dc.identifier.citationFischer e Silva, R.; Carpenter, P. M. High Throughput and Low Latency on Hadoop Clusters Using Explicit Congestion Notification: The Untold Truth. A: "2017 IEEE International Conference on Cluster Computing (CLUSTER)". IEEE, 2017, p. 349-353.
dc.identifier.isbn978-1-5386-2326-8
dc.identifier.urihttp://hdl.handle.net/2117/108637
dc.description.abstractVarious extensions of TCP/IP have been proposed to reduce network latency; examples include Explicit Congestion Notification (ECN), Data Center TCP (DCTCP) and several proposals for Active Queue Management (AQM). Combining these techniques requires adjusting various parameters, and recent studies have found that it is difficult to do so while obtaining both high performance and low latency. This is especially true for mixed use data centres that host both latency-sensitive applications and high-throughput workloads such as Hadoop.This paper studies the difficulty in configuration, and characterises the problem as related to ACK packets. Such packets cannot be set as ECN Capable Transport (ECT), with the consequence that a disproportionate number of them are dropped. We explain how this behavior decreases throughput, and propose a small change to the way that non-ECT-capable packets are handled in the network switches. We demonstrate robust performance for modified AQMs on a Hadoop cluster, maintaining full throughput while reducing latency by 85%. We also demonstrate that commodity switches with shallow buffers are able to reach the same throughput as deeper buffer switches. Finally, we explain how both TCP-ECN and DCTCP can achieve the best performance using a simple marking scheme, in constrast to the current preference for relying on AQMs to mark packets.
dc.description.sponsorshipThe research leading to these results has received funding from the European Unions Seventh Framework Programme (FP7/2007–2013) under grant agreement number 610456 (Euroserver). The research was also supported by the Ministry of Economy and Competitiveness of Spain under the contracts TIN2012-34557 and TIN2015-65316-P, Generalitat de Catalunya (contracts 2014-SGR-1051 and 2014-SGR-1272), HiPEAC-3 Network of Excellence (ICT- 287759), and the Severo Ochoa Program (SEV-2011-00067) of the Spanish Government.
dc.language.isoeng
dc.publisherIEEE
dc.rightsAttribution-NonCommercial-NoDerivs 3.0 Spain
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/3.0/es/
dc.subjectÀrees temàtiques de la UPC::Enginyeria electrònica
dc.subject.lcshCluster analysis--Data processing
dc.subject.otherThroughput
dc.subject.otherProposals
dc.subject.otherPorts (Computers)
dc.subject.otherRuntime
dc.subject.otherRobustness
dc.subject.otherReceivers
dc.subject.otherHadoop
dc.subject.otherECN
dc.subject.otherDCTCP
dc.subject.otherThroughput
dc.subject.otherLatency
dc.titleHigh Throughput and Low Latency on Hadoop Clusters Using Explicit Congestion Notification: The Untold Truth
dc.typeConference lecture
dc.subject.lemacComputació distribuïda
dc.subject.lemacSupercomputadors
dc.identifier.doi10.1109/CLUSTER.2017.19
dc.description.peerreviewedPeer Reviewed
dc.relation.publisherversionhttp://ieeexplore.ieee.org/abstract/document/8048946/
dc.rights.accessOpen Access
dc.description.versionPostprint (author's final draft)
dc.relation.projectidinfo:eu-repo/grantAgreement/MINECO/1PE/TIN2015-65316-P
local.citation.publicationName2017 IEEE International Conference on Cluster Computing (CLUSTER)
local.citation.startingPage349
local.citation.endingPage353


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record

Attribution-NonCommercial-NoDerivs 3.0 Spain
Except where otherwise noted, content on this work is licensed under a Creative Commons license : Attribution-NonCommercial-NoDerivs 3.0 Spain