Use elem.send_keys for handle "Infinite Scroll" in page. Using Selenium PhantomJS in Python -


i need take elements identified xpath in "infinite scroll" web page this. problem when use selenium webdriver phantomjs takes links, first links loaded after page loaded. try increase time.sleep() or insert more of there in code not works. if use firefox webdriver works well.

it's possibile sove problem , improve code? maybe not using time event looking can tell if go down on page or not.

thanks , greetings 

import re import mechanize pydblite import base selenium import webdriver import platform import codecs import scrapy   import time selenium.webdriver.common.keys import keys  class getfrom(object):    def scrapying(self):     print platform.system()           #browser = webdriver.firefox()         browser = webdriver.phantomjs(executable_path='/usr/local/bin/node_modules/phantomjs/lib/phantom/bin/phantomjs')        browser.get("https://medium.com/top-100/december-2013")     time.sleep(5)      elem = browser.find_element_by_tag_name("body")      no_of_pagedowns = 200      while no_of_pagedowns:         elem.send_keys(keys.page_down)         time.sleep(0.02)         no_of_pagedowns-=1      #qui ci dovrebbe essere lo spider      post_elems = browser.find_elements_by_class_name("graf--h2")      #fine spider      post in post_elems:         print post.text      browser.quit()  myclassobject = getfrom() myclassobject.scrapying()

i modified code this:

import re import mechanize pydblite import base selenium import webdriver import platform import codecs import scrapy import time selenium.webdriver.common.keys import keys  class getfrom(object):    def scrapying(self):     print platform.system()      if platform.system()=="windows":         browser = webdriver.firefox()     else:         #browser = webdriver.firefox()         browser = webdriver.phantomjs(executable_path='/usr/local/bin/node_modules/phantomjs/lib/phantom/bin/phantomjs')        browser.get("https://medium.com/top-100/december-2013")     time.sleep(5)      elem = browser.find_element_by_tag_name("body")       mins = raw_input("for how many minutes want scrapy links? ")      print "start: "+ time.asctime( time.localtime(time.time()) )     timeout = time.time() + 60* int(mins) # 5 minutes      while true:         test = 0         elem.send_keys(keys.page_down)         time.sleep(0.5)         if test == 5 or time.time() > timeout:             print "end: "+ time.asctime( time.localtime(time.time()) )             break         test = test - 1      #qui ci dovrebbe essere lo spider      post_elems = browser.find_elements_by_class_name("graf--h2")      #fine spider      post in post_elems:         print post.text      browser.quit()    myclassobject = getfrom() myclassobject.scrapying()

now works fine. delete loop add time control handling possibility of gain few results or lots of then. must important thing use javascript function: browser.execute_script("window.scrollto(0, document.body.scrollheight);")

instead of send_keys


Comments

Post a Comment

Popular posts from this blog

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

java - WARN : org.springframework.web.servlet.PageNotFound - No mapping found for HTTP request with URI [/board/] in DispatcherServlet with name 'appServlet' -

this question has answer here: why spring mvc respond 404 , report “no mapping found http request uri […] in dispatcherservlet”? 3 answers i trying making simple bbs i'm encountering following error, warn : org.springframework.web.servlet.pagenotfound - no mapping found http request uri [/board/] in dispatcherservlet name 'appservlet' i have tried solve error many times, have failed. how can solve error? here web.xml, controllerand , servlet-context. my web.xml <?xml version="1.0" encoding="utf-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <filter> <filter-nam...
Read more

Ruby on Rails, ActiveRecord, Postgres, UTF-8 and ASCII-8BIT encodings -

weirdest error, who’s challenge try , me? spent hours yesterday on , it’s magic. can't believe happening , it's driving me crazy. btw, using: rails 4.2.3 ruby 2.2.3 pg 0.18 encoding.default_internal = encoding.default_external = utf-8 config.encoding = 'utf-8' it has postgres, activerecord , encodings! turns out we’ve been getting these errors whenever create new users on database special characters: encoding::undefinedconversionerror: "\xc3" ascii-8bit utf-8 \xc3 may vary depending on characters. weird part we’ve set since beginning utf-8. makes no sense , digged it, , used script: user.all.each |user| user.attributes.each |name, value| if value.is_a? string puts user.email + name.encoding.to_s + value.encoding.to_s end end end and output of 1 of users, users same: dr_lottahelp@blahblah.comutf-8utf-8 dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8asci...
Read more