Hadoop YARN clusters - Adding node at runtime -


i working on solution providing run-time ressources addition hadoop yarn cluster. purpose handle heavy peaks on our application.

i not expert , need in order approve / contest understand.

hadoop yarn

this application run in cluster-mode. provides ressource management (cpu & ram). spark application, example, ask job done. yarn handles request , provides executor computing on yarn cluster.

hdfs - data & executors

the datas not shared through executors, have stored in file system. in case : hdfs. means have run copy of spark streaming application in new server (hadoop node).

i not sure of this :

the yarn cluster , hdfs different, writing on hdfs won't write on new hadoop node local data (because not hdfs node).

as write on hdfs new data spark streaming application, creating new application should not problem.

  1. submit job yarn
    --- peak, resources needed
  2. instance new server

  3. install / configure hadoop & yarn, making slave

    • modifying hadoop/conf/slaves, adding it's ip adress (or dns name host file)

    • moddifying dfs.include , mapred.include

      on host machine :

    • yarn -refreshnodes
    • bin/hadoop dfsadmin -refreshnodes
    • bin/hadoop mradmin -refreshnodes

should work ? refreshqueues sounds not useful here seems take care of process queue.

i not sure if running job increase it's capacity. idea wait new ressources available , submit new job.

thanks help


Comments

Post a Comment

Popular posts from this blog

html - Outlook 2010 Anchor (url/address/link) -

i'm trying link specific pdf page msoutlook'10 or msword'10, #page=... anchor particularly useful , works fine elsewhere. unfortunately when click [ctr-k] in outlook or winword, links re-formatted assuming unc path. pdf file located on shared drive. using "file:" prefix links open pdf in adobe reader (outside of browser), , "http:" open inside browser. issue seem need open in browser "file:" prefix in order anchor work. know anchors work inside browser. outlook uses windows/explorer default program (based on file extension) seems open in adobe reader neglecting anchor. is aware of solution, link specific page or bookmark of shared-pdf ms outlook and/or winword link? common references: mailermailer.com/...anchor-tags-html-emails campaignmonitor.com/...anchor-links-in-email-newsletters/ stackoverflow.com/...anchor-in-pdf-document
Read more

javascript - Why does running this loop 9 times take 100x longer than running it 8 times? -

consider code: test = function() { } t = new test(); (var = 0; < 8; i++) { result = t instanceof test; } if change number of iterations 8 9 , loop take 100 times longer complete in latest version of firefox (41.0.1). tested on 2 different pcs , magic limit 8. here jsperf test used: http://jsperf.com/instanceof-8-times-vs-9-times does have idea why might happen? seems specific instanceof . not happen if else object, example check property. note: filed bugzilla bug this. jan de mooij mozilla team has posted details in bugzilla thread . here's simplistic interpretation of highly technical answers: in i < 8 case, firefox smart enough hoist result = t instanceof test; statement out of loop (according tests, doesn't seem omit altogether). in i < 9 case, apparently doesn't optimization . why? reason not clear, related fact 9 iterations threshold above function considered "hot" enough run through jit compiler. i < 8 cas...
Read more

Getting gateway time-out Rails app with Nginx + Puma running on Digital Ocean -

following this guide setup app on digital ocean droplet, not working. site not loading , it's throwing: "gateway time-out". the puma server works, since ps aux | grep puma prints: root 15878 0.0 0.4 376832 2168 ? sl 12:50 0:00 puma 2.13.4 (unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock) [20150930164940] root 15881 0.0 0.7 903676 3692 ? sl 12:50 0:06 puma: cluster worker 0: 15878 [20150930164940] root 24654 0.0 0.1 11748 916 pts/2 s+ 18:00 0:00 grep --color=auto puma my nginx config: user root; worker_processes 1; events { worker_connections 1024; } http { upstream puma { server unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock; } server { listen 80; server_name myapp.com; root /var/www/myapp/current/public; access_log /var/www/myapp/current/log/access.log; error_log /...
Read more