Apache Spark - accessing internal data on RDDs? -


i started doing amp-camp 5 exercises. tried following 2 scenarios:

scenario #1 

val pagecounts = sc.textfile("data/pagecounts") pagecounts.checkpoint pagecounts.count

scenario #2 

val pagecounts = sc.textfile("data/pagecounts") pagecounts.count

the total time show in spark shell application ui different both scenarios.
scenario #1 took 0.5 seconds, while scenario #2 took 0.2 s

in scenario #1, checkpoint command nothing, it's neither transformation nor action. it's saying once rdd materializes after action, go ahead , save disk. missing here?

questions:

  1. i understand scenario #1 taking more time, because rdd check-pointed (written disk). there way can know time taken checkpoint, total time?
    spark shell application ui shows following - scheduler delay, task deserialization time, gc time, result serialization time, getting result time. but, doesn't show breakdown checkpointing.

  2. is there way access above metrics e.g. scheduler delay, gc time , save them programmatically? want log of above metrics every action invoked on rdd.

  3. how can programmatically access following information:

    • size of rdd, when persisted disk on checkpointing?
    • how percentage of rdd in memory currently?
    • overall time taken computing rdd?

please let me know if need more information.

spark rest api provides asking for.

some examples;

how percentage of rdd in memory currently?

get /api/v1/applications/[app-id]/storage/rdd/0

will responded with:

{   "id" : 0,   "name" : "parallelcollectionrdd",   "numpartitions" : 2,   "numcachedpartitions" : 2,   "storagelevel" : "memory deserialized 1x replicated",   "memoryused" : 28000032,   "diskused" : 0,   "datadistribution" : [ {     "address" : "localhost:54984",     "memoryused" : 28000032,     "memoryremaining" : 527755733,     "diskused" : 0   } ],   "partitions" : [ {     "blockname" : "rdd_0_0",     "storagelevel" : "memory deserialized 1x replicated",     "memoryused" : 14000016,     "diskused" : 0,     "executors" : [ "localhost:54984" ]   }, {     "blockname" : "rdd_0_1",     "storagelevel" : "memory deserialized 1x replicated",     "memoryused" : 14000016,     "diskused" : 0,     "executors" : [ "localhost:54984" ]   } ] }

overall time taken computing rdd?

to compute rdd called either job, stage, or attempt. get /applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/tasksummary

will responded with:

{   "quantiles" : [ 0.05, 0.25, 0.5, 0.75, 0.95 ],   "executordeserializetime" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ],   "executorruntime" : [ 3.0, 3.0, 4.0, 4.0, 4.0 ],   "resultsize" : [ 1457.0, 1457.0, 1457.0, 1457.0, 1457.0 ],   "jvmgctime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],   "resultserializationtime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],   "memorybytesspilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],   "diskbytesspilled" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],   "shufflereadmetrics" : {     "readbytes" : [ 340.0, 340.0, 342.0, 342.0, 342.0 ],     "readrecords" : [ 10.0, 10.0, 10.0, 10.0, 10.0 ],     "remoteblocksfetched" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],     "localblocksfetched" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ],     "fetchwaittime" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],     "remotebytesread" : [ 0.0, 0.0, 0.0, 0.0, 0.0 ],     "totalblocksfetched" : [ 2.0, 2.0, 2.0, 2.0, 2.0 ]   } }

your question broad, hence not respond all. believe spark has reflect reflected rest api.


Comments

Post a Comment

Popular posts from this blog

html - Outlook 2010 Anchor (url/address/link) -

i'm trying link specific pdf page msoutlook'10 or msword'10, #page=... anchor particularly useful , works fine elsewhere. unfortunately when click [ctr-k] in outlook or winword, links re-formatted assuming unc path. pdf file located on shared drive. using "file:" prefix links open pdf in adobe reader (outside of browser), , "http:" open inside browser. issue seem need open in browser "file:" prefix in order anchor work. know anchors work inside browser. outlook uses windows/explorer default program (based on file extension) seems open in adobe reader neglecting anchor. is aware of solution, link specific page or bookmark of shared-pdf ms outlook and/or winword link? common references: mailermailer.com/...anchor-tags-html-emails campaignmonitor.com/...anchor-links-in-email-newsletters/ stackoverflow.com/...anchor-in-pdf-document
Read more

javascript - Why does running this loop 9 times take 100x longer than running it 8 times? -

consider code: test = function() { } t = new test(); (var = 0; < 8; i++) { result = t instanceof test; } if change number of iterations 8 9 , loop take 100 times longer complete in latest version of firefox (41.0.1). tested on 2 different pcs , magic limit 8. here jsperf test used: http://jsperf.com/instanceof-8-times-vs-9-times does have idea why might happen? seems specific instanceof . not happen if else object, example check property. note: filed bugzilla bug this. jan de mooij mozilla team has posted details in bugzilla thread . here's simplistic interpretation of highly technical answers: in i < 8 case, firefox smart enough hoist result = t instanceof test; statement out of loop (according tests, doesn't seem omit altogether). in i < 9 case, apparently doesn't optimization . why? reason not clear, related fact 9 iterations threshold above function considered "hot" enough run through jit compiler. i < 8 cas...
Read more

Getting gateway time-out Rails app with Nginx + Puma running on Digital Ocean -

following this guide setup app on digital ocean droplet, not working. site not loading , it's throwing: "gateway time-out". the puma server works, since ps aux | grep puma prints: root 15878 0.0 0.4 376832 2168 ? sl 12:50 0:00 puma 2.13.4 (unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock) [20150930164940] root 15881 0.0 0.7 903676 3692 ? sl 12:50 0:06 puma: cluster worker 0: 15878 [20150930164940] root 24654 0.0 0.1 11748 916 pts/2 s+ 18:00 0:00 grep --color=auto puma my nginx config: user root; worker_processes 1; events { worker_connections 1024; } http { upstream puma { server unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock; } server { listen 80; server_name myapp.com; root /var/www/myapp/current/public; access_log /var/www/myapp/current/log/access.log; error_log /...
Read more