r - How to load csv file into SparkR on RStudio? -


how load csv file sparkr on rstudio? below steps had perform run sparkr on rstudio. have used read.df read .csv not sure how else write this. not sure if step considered create rdds. 

#set sys environment variables sys.setenv(spark_home = "c:/users/desktop/spark/spark-1.4.1-bin-hadoop2.6") .libpaths(c(file.path(sys.getenv("spark_home"), "r", "lib"), .libpaths()))  #sys.setenv('sparkr_submit_args'='"--packages" "com.databricks:spark-csv_2.10:1.0.3" "sparkr-shell"')  #load libraries library(sparkr) library(magrittr)  sc <- sparkr.init(master="local") sc <- sparkr.init() sc <- sparkr.init(sparkpackages="com.databricks:spark-csv_2.11:1.0.3") sqlcontext <- sparkrsql.init(sc)  data <- read.df(sqlcontext, "c:/users/desktop/datasets/hello_world.csv", "com.databricks.spark.csv", header="true")

i getting error: 

error in writejobj(con, object) : invalid jobj 1

spark 2.0.0+:

you can use csv data source:

loaddf(sqlcontext, path="some_path", source="csv", header="true")

without loading spark-csv.

original answer:

as far can tell you're using wrong version of spark-csv. pre-built versions of spark using scala 2.10, you're using spark csv scala 2.11. try instead:

sc <- sparkr.init(sparkpackages="com.databricks:spark-csv_2.10:1.2.0")

Comments

Post a Comment

Popular posts from this blog

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

java - WARN : org.springframework.web.servlet.PageNotFound - No mapping found for HTTP request with URI [/board/] in DispatcherServlet with name 'appServlet' -

this question has answer here: why spring mvc respond 404 , report “no mapping found http request uri […] in dispatcherservlet”? 3 answers i trying making simple bbs i'm encountering following error, warn : org.springframework.web.servlet.pagenotfound - no mapping found http request uri [/board/] in dispatcherservlet name 'appservlet' i have tried solve error many times, have failed. how can solve error? here web.xml, controllerand , servlet-context. my web.xml <?xml version="1.0" encoding="utf-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <filter> <filter-nam...
Read more

Ruby on Rails, ActiveRecord, Postgres, UTF-8 and ASCII-8BIT encodings -

weirdest error, who’s challenge try , me? spent hours yesterday on , it’s magic. can't believe happening , it's driving me crazy. btw, using: rails 4.2.3 ruby 2.2.3 pg 0.18 encoding.default_internal = encoding.default_external = utf-8 config.encoding = 'utf-8' it has postgres, activerecord , encodings! turns out we’ve been getting these errors whenever create new users on database special characters: encoding::undefinedconversionerror: "\xc3" ascii-8bit utf-8 \xc3 may vary depending on characters. weird part we’ve set since beginning utf-8. makes no sense , digged it, , used script: user.all.each |user| user.attributes.each |name, value| if value.is_a? string puts user.email + name.encoding.to_s + value.encoding.to_s end end end and output of 1 of users, users same: dr_lottahelp@blahblah.comutf-8utf-8 dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8asci...
Read more