python - Reduce job in Spark by reduceByKey() or other functions? -


given following list:

[(0, [135, 2]), (0, [2409, 1]), (0, [12846, 2]), (1, [13840, 2]), ...]

i need output list of first elements of list-value (i.e., 135, 2409, 12846 key 0 , 13840 key 1) each key if second element of list-value (i.e., 2, 1, 2 0 , 2 1) greater or equal value (let's 2). instance, in particular case output should be:

[(0, [135, 12846]), (1, [13840]), ...]

the tuple (0, [2409, 1]) discarded because 1 < 2.

i've achieved applying groupbykey(), mapvalues(list) , final map function, groupbykey() less efficient reduce functions.

is possible achieving task using reducebykey() or combinebykey() function?

the answer yes :) can achieve same reducebykey groupbykey. in fact, reducebykey should favoured performs map side reduce before shuffling data.

a solution using reducebykey (in scala, i'm sure point, , can convert python if prefer):

val rdd = sc.parallelize(list((0, list(135, 2)), (0, list(2409, 1)), (0, list(12846, 2)), (1, list(13840, 2)))) rdd.mapvalues(v => if(v(1) >= 2) list(v(0)) else list.empty)    .reducebykey(_++_)

Comments

Post a Comment

Popular posts from this blog

java - WARN : org.springframework.web.servlet.PageNotFound - No mapping found for HTTP request with URI [/board/] in DispatcherServlet with name 'appServlet' -

this question has answer here: why spring mvc respond 404 , report “no mapping found http request uri […] in dispatcherservlet”? 3 answers i trying making simple bbs i'm encountering following error, warn : org.springframework.web.servlet.pagenotfound - no mapping found http request uri [/board/] in dispatcherservlet name 'appservlet' i have tried solve error many times, have failed. how can solve error? here web.xml, controllerand , servlet-context. my web.xml <?xml version="1.0" encoding="utf-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <filter> <filter-nam...
Read more

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

android - How to create dynamically Fragment pager adapter -

i unable create dynamic layout each tabs, able create tabs dynamically constructor(public mypageradapter(fragmentmanager fm , int nooftabs)), coudn't inflate view each tabs creating each fragment should dyanmic. public class mypageradapter extends fragmentpageradapter { private final string[] titles = { "categories", "home", "top paid", "top free", "top grossing", "top new paid", "top new free", "trending" }; int nooftabs; **public mypageradapter(fragmentmanager fm , int nooftabs) { super(fm); this.nooftabs = nooftabs; }** /*public mypageradapter(fragmentmanager fm) { super(fm); }*/ @override public charsequence getpagetitle(int position) { return titles[position]; } @override public int getcount() { ...
Read more