Power By 专否社区

Search HTML line by line with regex in Python

Go To StackoverFlow.com

0

I'm attempting to create a dictionary of hours based off of this calendar: http://disneyworld.disney.go.com/parks/magic-kingdom/calendar/

<td class="first"> <div class="dayContainer">
      <a href="/parks/magic-kingdom/calendardayview/?asmbly_day=20120401"> 
         <p class="day"> 1
         </p> <p class="moreLink">Park Hours<br />8:00 AM - 12:00 AM<br /><br/>Extra Magic Hours<br />7:00 AM - 8:00 AM<br /><br/>Extra Magic Hours<br />12:00 AM - 3:00 AM<br /><br/>
         </p> 
      </a> 
   </div>
</td>

Each of the calendar entries are on a single line, so I figured it would be best to just go through the HTML line by line, and if that line contains hours, add those hours to a dictionary for the corresponding date (some days have multiple hour entries).

import urllib
import re
source = urllib.urlopen('http://disneyworld.disney.go.com/parks/magic-kingdom/c\
alendar/')
page = source.read()
prkhrs = {}

def main():
    parsehours()

def parsehours():
    #look for #:## AM - #:## PM                                                 
    date = r'201204\d{02}'
    hours = r'\d:0{2}\s\w{2}\s-\s\d:0{2}\s\w{2}'
    #go through page line by line                                               
    for line in page:
        times = re.findall(hours, line)
        dates = re.search(date, line)
        if dates:
            start = dates.start()
            end = dates.end()
            curdate = line[start:end]
        #if #:## - #:## is found, a date has been found                         
        if times:
            #create dictionary from date, stores hours in variable              
            #extra magic hours(emh) are stored in same format.                  
            #if entry has 2/3 hour listings, those listings are emh             
            prkhrs[curdate]['hours'] = times
    #just print hours for now. will change later                                
    print prkhrs

The problem I encounter is that when I put 'print line' inside the for loop that goes through the page, it prints it out a character at a time, which I'm assuming is what's messing things up.

Right now, the 'print prkhrs' just prints nothing, but using re.findall for both the dates and the hours prints out the correct times, so I know the regex works. Any suggestions on how I can get it to work?

2012-04-04 21:46
by westbyb
Parse the HTML - don't regex i - zellio 2012-04-04 21:48
@Mimisbrunnr, This doesn't appear to be using regex to parse HTML despite the title and tags - aaronasterling 2012-04-04 21:52
This looks like a job for BeautifulSoupJoel Cornett 2012-04-04 21:54
@aaronasterling - westbyb is parsing data out of the document via use of regex. He may not be directly parsing the HTML it self but it would be much easier to turn the HTML into a representative object structure and pull the data out usefully from there - zellio 2012-04-04 21:56
@Mimisbrunnr - how would you recommend I go about doing it that way? Regex is really the only way I know about going this, at least for now - westbyb 2012-04-04 22:07
@Mimisbrunnr Look at the input. One would get down to a span with br tags interspersing the useful data from the useless data. One would use a date regex to catch that anyways. I don't see a parser avoiding the use of regex here and one might as well use the regex to begin with. Were the regex such that it could possibly be confused by the presence of HTML then I would council using a parser to strip the tags off and then applying the regex. In this case, a parser seems like a useless level of indirection. FWIW, I'm not a huge regex fan and would rather quit before I parsed HTML with regex - aaronasterling 2012-04-04 22:23
@westbyb. Debate about regex here aside, I would surely learn to use an HTML parser if you plan on doing a lot of scraping. It's too big for a comment but google something like "parsing HTML with python" for a start and ask questions from there - aaronasterling 2012-04-04 22:25

6

Change page = source.read() to page = source.readlines()

source.read() returns the whole page as one big string. Iterating over a string (as when you do for line in page) returns one character at a time. Just because your variables are called line and page doesn't mean Python knows what you want.

source.readlines() returns a list of strings, each of which is a line from the page.

2012-04-04 21:49
by Whatang
FWIW, I agree with the commenters on the question that rather than using a regex in this way, you'd be better off parsing the HTML properly. But, for the question you actually asked this is a valid answer - Whatang 2012-04-04 22:02
That did it! I ran into some other issues, but that fixed the problem at hand. Thanks - westbyb 2012-04-04 22:06
Mind.Blown!!! Thank you!!!!! - Jose187 2012-09-30 08:56
Ads