azure - How Many Hive Dynamic Partitions are Needed? -


i running large job consolidates 55 streams (tags) of samples (one sample per record) @ irregular times on 2 years 15-minute averages. there 1.1 billion records in 23k streams in raw dataset, , these 55 streams make 33 million of records. calculated 15-minute index , grouping average value, seem have exceeded max dynamic partitions on hive job in spite of cranking way 20k. can increase further suppose, takes awhile fail (about 6 hours, although reduced 2 reducing number of streams consider), , don’t know how calculate how many need.

here code:

set hive.exec.dynamic.partition = true;  set hive.exec.dynamic.partition.mode = nonstrict;  set hive.exec.max.dynamic.partitions=50000; set hive.exec.max.dynamic.partitions.pernode=20000;    drop table if exists sensor_part_qhr;    create table sensor_part_qhr (     tag  string,     tag0 string,     tag1 string,     tagn_1  string,     tagn  string,      timestamp  string,     unixtime int,     qqfr2013 int,      quality  int,     count  int,     stdev  double,     value    double )   partitioned (bld string);  insert table sensor_part_qhr partition (bld)  select  tag,         min(tag),          min(tag0),          min(tag1),          min(tagn_1),          min(tagn),          min(timestamp),         min(unixtime),           qqfr2013,          min(quality),     count(value),     stddev_samp(value),          avg(value)    sensor_part_subset        tag1='energy' group tag,qqfr2013;

and here error message:

    error during job, obtaining debugging information...     examining task id: task_1442824943639_0044_m_000008 (and more) job job_1442824943639_0044     examining task id: task_1442824943639_0044_r_000000 (and more) job job_1442824943639_0044      task failures(4):      -----     task id:       task_1442824943639_0044_r_000000      url:       http://headnodehost:9014/taskdetails.jsp?jobid=job_1442824943639_0044&tipid=task_1442824943639_0044_r_000000     -----     diagnostic messages task:     error: java.lang.runtimeexception: org.apache.hadoop.hive.ql.metadata.hivefatalexception: [error 20004]: fatal error occurred when node tried create many dynamic partitions. maximum number of dynamic partitions controlled hive.exec.max.dynamic.partitions , hive.exec.max.dynamic.partitions.pernode. maximum set to: 20000         @ org.apache.hadoop.hive.ql.exec.mr.execreducer.reduce(execreducer.java:283)         @ org.apache.hadoop.mapred.reducetask.runoldreducer(reducetask.java:444)         @ org.apache.hadoop.mapred.reducetask.run(reducetask.java:392)         @ org.apache.hadoop.mapred.yarnchild$2.run(yarnchild.java:168)         @ java.security.accesscontroller.doprivileged(native method)         @ javax.security.auth.subject.doas(subject.java:415)         @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1594)         @ org.apache.hadoop.mapred.yarnchild.main(yarnchild.java:163)     caused by: org.apache.hadoop.hive.ql.metadata.hivefatalexception:      [error 20004]: fatal error occurred when node tried create many dynamic partitions.      maximum number of dynamic partitions controlled hive.exec.max.dynamic.partitions , hive.exec.max.dynamic.partitions.pernode.      maximum set to: 20000          @ org.apache.hadoop.hive.ql.exec.filesinkoperator.getdynoutpaths(filesinkoperator.java:747)         @ org.apache.hadoop.hive.ql.exec.filesinkoperator.startgroup(filesinkoperator.java:829)         @ org.apache.hadoop.hive.ql.exec.operator.defaultstartgroup(operator.java:498)         @ org.apache.hadoop.hive.ql.exec.operator.startgroup(operator.java:521)         @ org.apache.hadoop.hive.ql.exec.mr.execreducer.reduce(execreducer.java:232)         ... 7 more      container killed applicationmaster.     container killed on request. exit code 137     container exited non-zero exit code 137       failed: execution error, return code 2 org.apache.hadoop.hive.ql.exec.mr.mapredtask     mapreduce jobs launched:      job 0: map: 520  reduce: 140   cumulative cpu: 7409.394 sec   hdfs read: 0 hdfs write: 393345977 success     job 1: map: 9  reduce: 1   cumulative cpu: 87.201 sec   hdfs read: 393359417 hdfs write: 0 fail     total mapreduce cpu time spent: 0 days 2 hours 4 minutes 56 seconds 595 msec

can give ideas how calculate how many of these dynamic nodes might need job this?

or maybe should doing differently? running hive 0.13 way on azure hdinsight.

update:

  • corrected of numbers above.
  • reduced 3 streams operating on 211k records , succeeded.
  • started experimenting, reduced partitions per node 5k, , 1k, , still succeeded.

so not blocked anymore, thinking have needed millions of nodes whole dataset in 1 go (which wanted do).

dynamic partition columns must specified last among columns in select statement during insertion in sensor_part_qhr.


Comments

Post a Comment

Popular posts from this blog

html - Outlook 2010 Anchor (url/address/link) -

i'm trying link specific pdf page msoutlook'10 or msword'10, #page=... anchor particularly useful , works fine elsewhere. unfortunately when click [ctr-k] in outlook or winword, links re-formatted assuming unc path. pdf file located on shared drive. using "file:" prefix links open pdf in adobe reader (outside of browser), , "http:" open inside browser. issue seem need open in browser "file:" prefix in order anchor work. know anchors work inside browser. outlook uses windows/explorer default program (based on file extension) seems open in adobe reader neglecting anchor. is aware of solution, link specific page or bookmark of shared-pdf ms outlook and/or winword link? common references: mailermailer.com/...anchor-tags-html-emails campaignmonitor.com/...anchor-links-in-email-newsletters/ stackoverflow.com/...anchor-in-pdf-document
Read more

javascript - Why does running this loop 9 times take 100x longer than running it 8 times? -

consider code: test = function() { } t = new test(); (var = 0; < 8; i++) { result = t instanceof test; } if change number of iterations 8 9 , loop take 100 times longer complete in latest version of firefox (41.0.1). tested on 2 different pcs , magic limit 8. here jsperf test used: http://jsperf.com/instanceof-8-times-vs-9-times does have idea why might happen? seems specific instanceof . not happen if else object, example check property. note: filed bugzilla bug this. jan de mooij mozilla team has posted details in bugzilla thread . here's simplistic interpretation of highly technical answers: in i < 8 case, firefox smart enough hoist result = t instanceof test; statement out of loop (according tests, doesn't seem omit altogether). in i < 9 case, apparently doesn't optimization . why? reason not clear, related fact 9 iterations threshold above function considered "hot" enough run through jit compiler. i < 8 cas...
Read more

Getting gateway time-out Rails app with Nginx + Puma running on Digital Ocean -

following this guide setup app on digital ocean droplet, not working. site not loading , it's throwing: "gateway time-out". the puma server works, since ps aux | grep puma prints: root 15878 0.0 0.4 376832 2168 ? sl 12:50 0:00 puma 2.13.4 (unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock) [20150930164940] root 15881 0.0 0.7 903676 3692 ? sl 12:50 0:06 puma: cluster worker 0: 15878 [20150930164940] root 24654 0.0 0.1 11748 916 pts/2 s+ 18:00 0:00 grep --color=auto puma my nginx config: user root; worker_processes 1; events { worker_connections 1024; } http { upstream puma { server unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock; } server { listen 80; server_name myapp.com; root /var/www/myapp/current/public; access_log /var/www/myapp/current/log/access.log; error_log /...
Read more