Apache Hadoop YARN – ResourceManager
As previously described, ResourceManager (RM) is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NodeManagers (NMs) and the per-application ApplicationMasters (AMs).
- NodeManagers take instructions from the ResourceManager and manage resources available on a single node.
- ApplicationMasters are responsible for negotiating resources with the ResourceManager and for working with the NodeManagers to start the containers.
ResourceManager Components
The ResourceManager has the following components (see the figure above):
-
Components interfacing RM to the clients:
- ClientService: The client interface to the Resource Manager. This component handles all the RPC interfaces to the RM from the clients including operations like application submission, application termination, obtaining queue information, cluster statistics etc.
- AdminService: To make sure that admin requests don’t get starved due to the normal users’ requests and to give the operators’ commands the higher priority, all the admin operations like refreshing node-list, the queues’ configuration etc. are served via this separate interface.
-
Components connecting RM to the nodes:
- ResourceTrackerService: This is the component that responds to RPCs from all the nodes. It is responsible for registration of new nodes, rejecting requests from any invalid/decommissioned nodes, obtain node-heartbeats and forward them over to the YarnScheduler. It works closely with NMLivelinessMonitor and NodesListManager described below.
- NMLivelinessMonitor: To keep track of live nodes and specifically note down the dead nodes, this component keeps track of each node’s its last heartbeat time. Any node that doesn’t heartbeat within a configured interval of time, by default 10 minutes, is deemed dead and is expired by the RM. All the containers currently running on an expired node are marked as dead and no new containers are scheduling on such node.
-
NodesListManager: A collection of valid and excluded nodes. Responsible for reading the host configuration files specified via
yarn.resourcemanager.nodes.include-path
andyarn.resourcemanager.nodes.exclude-path
and seeding the initial list of nodes based on those files. Also keeps track of nodes that are decommissioned as time progresses.
-
Components interacting with the per-application AMs:
- ApplicationMasterService: This is the component that responds to RPCs from all the AMs. It is responsible for registration of new AMs, termination/unregister-requests from any finishing AMs, obtaining container-allocation & deallocation requests from all running AMs and forward them over to the YarnScheduler. This works closely with AMLivelinessMonitor described below.
- AMLivelinessMonitor: To help manage the list of live AMs and dead/non-responding AMs, this component keeps track of each AM and its last heartbeat time. Any AM that doesn’t heartbeat within a configured interval of time, by default 10 minutes, is deemed dead and is expired by the RM. All the containers currently running/allocated to an AM that gets expired are marked as dead. RM schedules the same AM to run on a new container, allowing up to a maximum of 4 such attempts by default.
-
The core of the ResourceManager – the scheduler and related components:
- ApplicationsManager: Responsible for maintaining a collection of submitted applications. Also keeps a cache of completed applications so as to serve users’ requests via web UI or command line long after the applications in question finished.
- ApplicationACLsManager: RM needs to gate the user facing APIs like the client and admin requests to be accessible only to authorized users. This component maintains the ACLs lists per application and enforces them whenever an request like killing an application, viewing an application status is received.
- ApplicationMasterLauncher: Maintains a thread-pool to launch AMs of newly submitted applications as well as applications whose previous AM attempts exited due to some reason. Also responsible for cleaning up the AM when an application has finished normally or forcefully terminated.
- YarnScheduler: The Scheduler is responsible for allocating resources to the various running applications subject to constraints of capacities, queues etc. It performs its scheduling function based on the resource requirements of the applications such as memory, CPU, disk, network etc. Currently, only memory is supported and support for CPU is close to completion.
- ContainerAllocationExpirer: This component is in charge of ensuring that all allocated containers are used by AMs and subsequently launched on the correspond NMs. AMs run as untrusted user code and can potentially hold on to allocations without using them, and as such can cause cluster under-utilization. To address this, ContainerAllocationExpirer maintains the list of allocated containers that are still not used on the corresponding NMs. For any container, if the corresponding NM doesn’t report to the RM that the container has started running within a configured interval of time, by default 10 minutes, the container is deemed as dead and is expired by the RM.
-
TokenSecretManagers (for security):ResourceManager has a collection of SecretManagers which are charged with managing tokens, secret-keys that are used to authenticate/authorize requests on various RPC interfaces. A future post on YARN security will cover a more detailed descriptions of the tokens, secret-keys and the secret-managers but a brief summary follows:
- ApplicationTokenSecretManager: To avoid arbitrary processes from sending RM scheduling requests, RM uses the per-application tokens called ApplicationTokens. This component saves each token locally in memory till application finishes and uses it to authenticate any request coming from a valid AM process.
- ContainerTokenSecretManager: SecretManager for ContainerTokens that are special tokens issued by RM to an AM for a container on a specific node. ContainerTokens are used by AMs to create a connection to the corresponding NM where the container is allocated. This component is RM-specific, keeps track of the underlying master and secret-keys and rolls the keys every so often.
- RMDelegationTokenSecretManager: A ResourceManager specific delegation-token secret-manager. It is responsible for generating delegation tokens to clients which can be passed on to unauthenticated processes that wish to be able to talk to RM.
- DelegationTokenRenewer: In secure mode, RM is Kerberos authenticated and so provides the service of renewing file-system tokens on behalf of the applications. This component renews tokens of submitted applications as long as the application runs and till the tokens can no longer be renewed.
Conclusion
In YARN, the ResourceManager is primarily limited to scheduling i.e. only arbitrating available resources in the system among the competing applications and not concerning itself with per-application state management. Because of this clear separation of responsibilities coupled with the modularity described above, and with the powerful scheduler API discussed in the previous post, RM is able to address the most important design requirements – scalability, support for alternate programming paradigms.
To allow for different policy constraints, the scheduler described above in the RM is pluggable and allows for different algorithms. In a future post of this series, we will dig deeper into various features of CapacityScheduler that schedules containers based on capacity guarantees and queues.
The next post will dive into details of the NodeManager, the component responsible for managing the containers’ life cycle and much more.
相关推荐
Apache Hadoop YARN.pdf完整电子版
Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2 完整版哦,绝对清晰,不是扫描的mobi格式电子书,请使用电子书库calibre (http://calibre-ebook.com/download) 打开。
利用Hadoop YARN ResourceManager 未授权访问getshell工具以及WORD说明
Apache Hadoop YARN:Moving beyond MapReduce and Batch Processing with Apach 2 【yarn权威指南】
软件介绍: ...mobi格式的清晰版Apache Hadoop YARN,这个是高清版不是扫描版的,请使用calibre电子书阅读器进行阅读,具体效果可以查看上面的书籍截图。本书深入讲解hadoop2的原理,很详细值得一读。
赠送jar包:hadoop-yarn-server-...标签:server、apache、resourcemanager、hadoop、yarn、jar包、java、API文档、中文版; 使用方法:解压翻译后的API文档,用浏览器打开“index.html”文件,即可纵览文档内容。 人性
《Apache Hadoop YARN》,全名Apache Hadoop YARN: Moving beyond MapReduce and Batch Processing with Apache Hadoop 2 (Addison-Wesley Data & Analytics Series) 这本书是2014年3月31号出版的,是基于Hadoop 2...
2013-Apache Hadoop YARN_ Yet Another Resource Negotiator.pdf
Apache Hadoop TM YARN Moving beyond MapReduce and Batch Processing with Apache HadoopTM 2 全本,不是sample。
Maven坐标:org.apache.hadoop:hadoop-yarn-api:2.5.1; 标签:apache、hadoop、api、yarn、jar包、java、API文档、中文版; 使用方法:解压翻译后的API文档,用浏览器打开“index.html”文件,即可纵览文档内容。 ...
apache hadoop tutorial
Apache Hadoop YARN is the modern distributed operating system for big data applications. It morphed the Hadoop compute layer to be a common resource-management platform that can host a wide variety of...
英文版,中文书名:Hadoop YARN权威指南。高清完美原书PDF版本,非MOBI版。
Maven坐标:org.apache.hadoop:hadoop-yarn-client:2.6.5; 标签:apache、client、hadoop、yarn、jar包、java、中文文档; 使用方法:解压翻译后的API文档,用浏览器打开“index.html”文件,即可纵览文档内容。 ...
hadoop2.7.4安装包补丁包,解决yarn定时调度启动问题!!
Spring Data for Apache Hadoop API。 Spring Data for Apache Hadoop 开发文档
详细介绍Hadoop Yarn的架构以及如何构建Hadoop Yarn。详解介绍如何进行资源调优。