python - Beautiful Soup is Missing Tables from Wikipedia -


i trying la liga league table wikipedia, can't seem use find_all table i'm trying scrape. moreover, exact same code wrote scrapes epl data wikipedia fine...

the full html here: view-source:https://en.wikipedia.org/wiki/2015%e2%80%9316_la_liga

the part in question here:

<h2><span class="mw-headline" id="league_table">league table</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=2015%e2%80%9316_la_liga&amp;action=edit&amp;section=6" title="edit section: league table">edit</a><span class="mw-editsection-bracket">]</span></span></h2>  <h3><span class="mw-headline" id="standings">standings</span><span class="mw-editsection"><span class="mw-editsection-bracket">[</span><a href="/w/index.php?title=2015%e2%80%9316_la_liga&amp;action=edit&amp;section=7" title="edit section: standings">edit</a><span class="mw-editsection-bracket">]</span></span></h3>  <table class="wikitable" style="text-align:center;">    <tr>      <th scope="col" width="28"><abbr title="position">pos</abbr>      </th>      <th scope="col" width="190">team        <div class="plainlinks hlist navbar mini" style="float:right">          <ul>            <li class="nv-view"><a href="/wiki/template:2015%e2%80%9316_la_liga_table" title="template:2015–16 la liga table"><span title="view template">v</span></a>            </li>            <li class="nv-talk"><a href="/wiki/template_talk:2015%e2%80%9316_la_liga_table" title="template talk:2015–16 la liga table"><span title="discuss template">t</span></a>            </li>            <li class="nv-edit"><a class="external text" href="//en.wikipedia.org/w/index.php?title=template:2015%e2%80%9316_la_liga_table&amp;action=edit"><span title="edit template">e</span></a>            </li>          </ul>        </div>      </th>

this how request page , cleansing of code before try find of tables:

soup = beautifulsoup(requests.get("https://en.wikipedia.org/wiki/2015-16_la_liga").text, "html.parser") superscript in soup.find_all("sup"):     superscript.decompose() print len(soup.find_all("table", attrs={"class": "wikitable"}))

however getting length of 2 when looking @ page html, should getting @ least 14 tables attributes...

i have no idea go here, appreciated

--edit--

input/output output of soup shows wikitable still there...

everything work fine.

pyquery version

from pyquery import pyquery  pq = pyquery(url="https://en.wikipedia.org/wiki/2015-16_la_liga") all_tables = pq(".wikitable") print len(all_tables)

beautifulsoup version

__author__ = "leonard richardson (leonardr@segfault.org)" __version__ = "4.3.2" __copyright__ = "copyright (c) 2004-2013 leonard richardson" __license__ = "mit"  bs4 import beautifulsoup import requests    soup = beautifulsoup(requests.get("https://en.wikipedia.org/wiki/2015-16_la_liga").text, "html.parser") superscript in soup.find_all("sup"):     superscript.decompose() print len(soup.find_all("table", attrs={"class": "wikitable"}))

return 13 both version

maybe should install 4.3.2 version of bs or use pyquery?


Comments

Post a Comment

Popular posts from this blog

1111. appearing after print sequence - php -

morning, i have code takes string of shoe sizes, explodes , puts in table. code works , info displayed correctly on page after table there few 1's after table. i'm confused this. so table looks (in columns of 3 on webpage) available in following sizes: 3 4 5 6 9 10 111 here's codes, purpose of example $x = 3|4|6|10|9 function get_sizes_pipe($x) { $counter = 0; $splits = explode("|",$x); asort($splits); $x = print "<table class='greentable'>"; $x .= print "<thead><tr><th scope='col' colspan='3' abbr='starter'>available in following sizes:</th></tr></thead><tbody><tr>"; foreach ($splits $split) { $entry = print "<td>".$split."</td>"; if($counter == 2){ $entry .= print "</tr><tr>"; $counter =0; } else { $counter++; ...
Read more

java - WARN : org.springframework.web.servlet.PageNotFound - No mapping found for HTTP request with URI [/board/] in DispatcherServlet with name 'appServlet' -

this question has answer here: why spring mvc respond 404 , report “no mapping found http request uri […] in dispatcherservlet”? 3 answers i trying making simple bbs i'm encountering following error, warn : org.springframework.web.servlet.pagenotfound - no mapping found http request uri [/board/] in dispatcherservlet name 'appservlet' i have tried solve error many times, have failed. how can solve error? here web.xml, controllerand , servlet-context. my web.xml <?xml version="1.0" encoding="utf-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"> <filter> <filter-nam...
Read more

Ruby on Rails, ActiveRecord, Postgres, UTF-8 and ASCII-8BIT encodings -

weirdest error, who’s challenge try , me? spent hours yesterday on , it’s magic. can't believe happening , it's driving me crazy. btw, using: rails 4.2.3 ruby 2.2.3 pg 0.18 encoding.default_internal = encoding.default_external = utf-8 config.encoding = 'utf-8' it has postgres, activerecord , encodings! turns out we’ve been getting these errors whenever create new users on database special characters: encoding::undefinedconversionerror: "\xc3" ascii-8bit utf-8 \xc3 may vary depending on characters. weird part we’ve set since beginning utf-8. makes no sense , digged it, , used script: user.all.each |user| user.attributes.each |name, value| if value.is_a? string puts user.email + name.encoding.to_s + value.encoding.to_s end end end and output of 1 of users, users same: dr_lottahelp@blahblah.comutf-8utf-8 dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8ascii-8bit dr_lottahelp@blahblah.comutf-8asci...
Read more