SQLITE_ERROR: Connection is closed when connecting from Spark via JDBC to SQLite database -


i using apache spark 1.5.1 , trying connect local sqlite database named clinton.db. creating data frame table of database works fine when operations on created object, error below says "sql error or missing database (connection closed)". funny thing result of operation nevertheless. idea can solve problem, i.e., avoid error?

start command spark-shell:

../spark/bin/spark-shell --master local[8] --jars ../libraries/sqlite-jdbc-3.8.11.1.jar --classpath ../libraries/sqlite-jdbc-3.8.11.1.jar

reading database:

val emails = sqlcontext.read.format("jdbc").options(map("url" -> "jdbc:sqlite:../data/clinton.sqlite", "dbtable" -> "emails")).load()

simple count (fails):

emails.count

error:

15/09/30 09:06:39 warn jdbcrdd: exception closing statement java.sql.sqlexception: [sqlite_error] sql error or missing database (connection closed) @ org.sqlite.core.db.newsqlexception(db.java:890) @ org.sqlite.core.corestatement.internalclose(corestatement.java:109) @ org.sqlite.jdbc3.jdbc3statement.close(jdbc3statement.java:35) @ org.apache.spark.sql.execution.datasources.jdbc.jdbcrdd$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$jdbcrdd$$anon$$close(jdbcrdd.scala:454) @ org.apache.spark.sql.execution.datasources.jdbc.jdbcrdd$$anon$1$$anonfun$8.apply(jdbcrdd.scala:358) @ org.apache.spark.sql.execution.datasources.jdbc.jdbcrdd$$anon$1$$anonfun$8.apply(jdbcrdd.scala:358) @ org.apache.spark.taskcontextimpl$$anon$1.ontaskcompletion(taskcontextimpl.scala:60) @ org.apache.spark.taskcontextimpl$$anonfun$marktaskcompleted$1.apply(taskcontextimpl.scala:79) @ org.apache.spark.taskcontextimpl$$anonfun$marktaskcompleted$1.apply(taskcontextimpl.scala:77) @ scala.collection.mutable.resizablearray$class.foreach(resizablearray.scala:59) @ scala.collection.mutable.arraybuffer.foreach(arraybuffer.scala:47) @ org.apache.spark.taskcontextimpl.marktaskcompleted(taskcontextimpl.scala:77) @ org.apache.spark.scheduler.task.run(task.scala:90) @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:214) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1142) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:617) @ java.lang.thread.run(thread.java:745) res1: long = 7945

i got same error today, , important line before exception:

15/11/30 12:13:02 info jdbc.jdbcrdd: closed connection

15/11/30 12:13:02 warn jdbc.jdbcrdd: exception closing statement java.sql.sqlexception: [sqlite_error] sql error or missing database (connection closed) @ org.sqlite.core.db.newsqlexception(db.java:890) @ org.sqlite.core.corestatement.internalclose(corestatement.java:109) @ org.sqlite.jdbc3.jdbc3statement.close(jdbc3statement.java:35) @ org.apache.spark.sql.execution.datasources.jdbc.jdbcrdd$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$jdbcrdd$$anon$$close(jdbcrdd.scala:454)

so spark succeeded close jdbc connection, , fails close jdbc statement

looking @ source, close() called twice:

line 358 (org.apache.spark.sql.execution.datasources.jdbc.jdbcrdd, spark 1.5.1)

context.addtaskcompletionlistener{ context => close() }

line 469

override def hasnext: boolean = {   if (!finished) {     if (!gotnext) {       nextvalue = getnext()       if (finished) {         close()       }       gotnext = true     }   }   !finished }

if @ close() method (line 443)

def close() {   if (closed) return

you can see checks variable closed, value never set true.

if see correctly, bug still in master. have filed bug report.


Comments

Post a Comment

Popular posts from this blog

html - Outlook 2010 Anchor (url/address/link) -

i'm trying link specific pdf page msoutlook'10 or msword'10, #page=... anchor particularly useful , works fine elsewhere. unfortunately when click [ctr-k] in outlook or winword, links re-formatted assuming unc path. pdf file located on shared drive. using "file:" prefix links open pdf in adobe reader (outside of browser), , "http:" open inside browser. issue seem need open in browser "file:" prefix in order anchor work. know anchors work inside browser. outlook uses windows/explorer default program (based on file extension) seems open in adobe reader neglecting anchor. is aware of solution, link specific page or bookmark of shared-pdf ms outlook and/or winword link? common references: mailermailer.com/...anchor-tags-html-emails campaignmonitor.com/...anchor-links-in-email-newsletters/ stackoverflow.com/...anchor-in-pdf-document
Read more

javascript - Why does running this loop 9 times take 100x longer than running it 8 times? -

consider code: test = function() { } t = new test(); (var = 0; < 8; i++) { result = t instanceof test; } if change number of iterations 8 9 , loop take 100 times longer complete in latest version of firefox (41.0.1). tested on 2 different pcs , magic limit 8. here jsperf test used: http://jsperf.com/instanceof-8-times-vs-9-times does have idea why might happen? seems specific instanceof . not happen if else object, example check property. note: filed bugzilla bug this. jan de mooij mozilla team has posted details in bugzilla thread . here's simplistic interpretation of highly technical answers: in i < 8 case, firefox smart enough hoist result = t instanceof test; statement out of loop (according tests, doesn't seem omit altogether). in i < 9 case, apparently doesn't optimization . why? reason not clear, related fact 9 iterations threshold above function considered "hot" enough run through jit compiler. i < 8 cas...
Read more

Getting gateway time-out Rails app with Nginx + Puma running on Digital Ocean -

following this guide setup app on digital ocean droplet, not working. site not loading , it's throwing: "gateway time-out". the puma server works, since ps aux | grep puma prints: root 15878 0.0 0.4 376832 2168 ? sl 12:50 0:00 puma 2.13.4 (unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock) [20150930164940] root 15881 0.0 0.7 903676 3692 ? sl 12:50 0:06 puma: cluster worker 0: 15878 [20150930164940] root 24654 0.0 0.1 11748 916 pts/2 s+ 18:00 0:00 grep --color=auto puma my nginx config: user root; worker_processes 1; events { worker_connections 1024; } http { upstream puma { server unix:///var/www/myapp/shared/tmp/sockets/myapp-puma.sock; } server { listen 80; server_name myapp.com; root /var/www/myapp/current/public; access_log /var/www/myapp/current/log/access.log; error_log /...
Read more