scala - Spark + Kafka integration - mapping of Kafka partitions to RDD partitions -


i have couple of basic questions related spark streaming

[please let me know if these questions have been answered in other posts - couldn't find any]:

(i) in spark streaming, number of partitions in rdd default equal number of workers?

(ii) in direct approach spark-kafka integration, number of rdd partitions created equal number of kafka partitions. right assume each rdd partition i mapped same worker node j in every batch of dstream? ie, mapping of partition worker node based solely on index of partition? example, partition 2 assigned worker 1 in 1 batch , worker 3 in another?

thanks in advance

i) default parallelism number of cores (or 8 mesos), number of partitions input stream implementation

ii) no, mapping of partition indexes worker nodes not deterministic. if you're running kafka on same nodes spark executors, preferred location run task on node of kafka leader partition. then, task may scheduled on node.


Comments

Post a Comment

Popular posts from this blog

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

node.js - Express and Redis - If session exists for this user, don't allow access -

i using redis , express this: // app.js var redis = require('redis'), redisclient = redis.createclient(), session = require('express-session'), redisstore = require('connect-redis')(session); session = session({ store: new redisstore({client: redisclient}), secret: 'secretkey', resave: true, saveuninitialized: true }) app.use(session); in index route file, set user session this: req.session.user = user; when user comes (or opens tab), how detect how many sessions having , block him access if has 1 active session? express-session working via cookies. had cookie named session sessionid. tabs inside browsed share cookies , can't rely on cookies if want strict working via 1 browser tab. you need create own sessions each opened browser tab. generating tabid @ client side , saving {userid, tabid} redis or can use websockets ( http://socket.io/ ) that, little overhead, doing want
Read more

excel - I can't get the attachement of the email PHP -

i have script connect imap , excel files. worked till today. 1 of our supplier script can't attached file. think there wrong decoding email. this code decoding if($attachments[$i]['is_attachment']) { $attachments[$i]['attachment'] = imap_fetchbody($this->_mbox,$email_number,$i); /* 4 = quoted-printable encoding */ if($flattenedparts[$i]->encoding == 3) { $attachments[$i]['attachment'] = base64_decode($attachments[$i]['attachment']); } elseif($flattenedparts[$i]->encoding == 4)/* 3 = base64 encoding */ { $attachments[$i]['attachment'] = quoted_printable_decode($attachments[$i]['attachment']); } } is there other ways decode email imap on php? the array of imap array ( [1] => stdclass object ( [type] => 0 [encoding] => 0 [ifsubtype] => 1 [subtype] => plain [ifdescription] => 0 [ifid] => 0 [lines] ...
Read more