This lecture is about reading XML.

So if you don't know about XML, it's

Extensible Markup Language, is what it stands for.

So it's frequently used to score structured data.

It's particularly widely used in internet applications.

So you'll see it a lot when you're doing

things like web scraping or trying to get data from

an internet API or trying to download data from say,

open data of websites, say like an open government website.

So, extracting HTML, XML is actually the basis

for most of the web-scraping that you'll see.

There are two components to an XML file.

There's the markup.

That's the labels that give the text structure.

So you can imagine if you just started typing, you

would end up with sort of an unstructured text file.

The markup is the way that you add labels so that the file ends being structured.

And then the content is the actual text that you type in,

in between sort of the labels that give the structure to the text.

So to be a little bit more concrete, we'll talk about tags, elements and attributes.

So tags correspond to the labels, that there are labels that are going to

be applying to particular parts of the text so that it will be structured.

So, there are start tags.

And so, start tags look like this.

So, they start with an open caret on one side and then they

have a, a phrase and then a closed caret on the other side.

So, you'll have that at the beginning of a certain part of the text.

So, that will start them, that text as a, a labelled section called section.

And then it ends with an end tag which looks like this.

It has the same phrase as before, but it

starts with a forward slash inside of the two carets.

And so the empty tags are tags that

correspond to lines where you don't necessarily need

both an open and then a close tag to deliminate a certain part of a text.

You might just want to denote one specific line, say, as a line break.

And so what you do is, you have the same sort of structure.

Only you have the forward slash just over here at the right end.

So elements are specific examples of tags.

So, for example, one element of the XML

markup could be, the tag greeting, with the end

tag greeting over here on the right hand side

and in between it the content, the hello world.

So, attributes are components of the label.

So, you can actually add to the tags components.

And so, for example, you might have an image tag.

So, what you might want to have corresponding to

that image is an attribute that's the source.

That's where the image is actually coming from.

And then an alternate phrase for that tag, and that could be instructor, for example.

So another example here is suppose you have a tag that's

a step tag, and you might have multiple steps, so tag

number step number one might be tagged with number equal 1

and then two and then this is number three step three.

And then it needs to end with an end tag just like it did before.

If you want to know a lot more about XML and

it's probably Probably a good idea before doing a whole bunch

more with this lecture, if you could go and see, the

Wikepedia entry on XML, which is actually quite comprehensive and nice.

So this is an example XML file.

It's a little bit hard to see, but you can see it okay if

you go actually to the web link here at the bottom of the page.

And so the idea here is you have a text file,

and that text file has lots of tags and content in it.

So for example here you have a tag that says food and then

here is a close tag for that tag that says food down here.

So you can see they're indented at the same level within this file.

And then for this particular food element you actually have a name, so there's

the open tag for name and there's the close tag for name, and this particular.

Element is named sort of Belgian Waffle, delicious.

And then there's other tags that correspond

to other components of this particular food element.

And then you go down and there's another

food elements with another names and so forth.

So that's what an xml file look at.

It looks like and I encourage you to go look the

the bigger version of that when you go to that web link.

So you can read the file into R, you can do this

with the XML package so we're loading the XML package all caps XML.

And then you give it the url.

So in this case, we're going to give it the url from

the previous XML file we were looking at on that previous screen.

And then we can use the function XML three parts to parse out that XML file.

So what this does is.

It loads the document into a memory, into a R memory in a way

that you can then parse it and get access to different parts of it.

So, within R, it's still a structured object, so we have to

be able to use different functions to access different parts of that object.

So, the first thing that we want to look at is the root node.

So, the root node you can get with XML root.

And so what that is, is it's sort of the wrapper,

you can think of it as the wrapper for the entire document.

And so if you execute this xml root command, you'll

have access to that particular element to that xml file.

And then if you want to get the name out, you can actually use the xmlname

applied to that node to get the name out, in this case, it's the breakfast menu.

You can also look at the names of the root node.

And so, when you do names of the root node what it's actually

telling you is what all the nested elements within that root node are.

So, the root node wraps the whole document

and the whole document is a breakfast menu.

And then there are five different breakfast items on this

menu, and each one is wrapped within a food element.

So, you get five food elements, if you look at names of the root node.

So,

the next thing that you could use is

actually directly access parts of the XML document.

And you can do it in a little bit in the same that way you access a list in R.

And so, for example, if you want to look at.

You, you have this root node element that we've extracted.

And we can look at the first component of that or the

first element of that and we do it by using double brackets here.

That's how you access the first element, say in a list.

And so, what you get out, actually, is the first food element, like this.

So it contains the food tags, and it also

contains all the information about the first food element.

Then, if you want to keep drilling down into smaller and smaller parts of

the XML document, you can do sub-setting, just like you do with lists.

So first, this exit rootNode and the first bracket one, will again give you this.

Element here with the entire food component and then if you

want to take the first element of that, you'll just repeat the

process again and then you'll be looking at just the name

the belgian waffles because that's the

first subcomponent of the first subcomponent.

You can actually programmatically extract different parts of the file.

So you can do that with the XML Xsupply command,

and so what you do is you pass that a

parsed XML object and then you tell it what function

you like to apply; so in this case XML value.

So what that's going to do is, it's going to

loop through all of the elements of the

root node and get the XML value And

by default this is going to do this recursively.

So, if you apply it in this way,

since rootNode contains the entire document, it's going to

go through and get every single value of

every sing tagged element in the entire document.

And so you just get a bunch of text all

strung together that's all the text that was in that document.

You might be a little bit more specific

and get a specific component of the document.

And you can do that using the XPath language, so this a whole other language

that you have to learn in addition to XML to be able to access the data.

But if you learn just a few of the

components then you'll be a long way along the way.

I'm going to talk a little bit about this, but

there's actually really nice set of lecture notes that are

available here that go into this in a little

bit more depth, if you find yourself a little bit.

Lost after this lecture.

So the first thing that you're going to be looking at is the top

level node of each element, and that you get with a forward slash node.

The node at any level is double slash node.

And then you can actually extract specific nodes with specific attributes.

And we'll talk about how to do that in the next couple of slides.

[SOUND] So the first thing that you're going to want to be able to do is

get the items on the menu and their prices and so the way that

we're going to do that is we're again

going to use this xpathSApply but now we're

going to be a little bit more targeted

about which data we're going to be extracting.

So we're giving you, again, [INAUDIBLE] the rootNode.

That's the entire document, and so then what we're going to be doing

is we're going to be using This xpack element here is //name.

And so, what that's going to do is it's

going to go through and get all of the nodes

that correspond to an element with title name and

then it's going to get the xmlvalue of those nodes.

So, what it's going to do is it's going to take

out basically all of the elements of that XML file that

are tagged with name and so you can see, you end

up with the names of all of the items on the menu.

You can also do the same thing.

You send the exact same rootNode to xpathSApply, only now you

can say I actually just want the nodes that correspond to price.

And again I want their xmlValue.

And so it will plot out the xmel values which the prices of the values and now

we extract from that xmel file the names of the menu items as well as their prices.

To give you an example of, it's a

little more complicated I am actually going to use

a website of the Baltimore Ravens which is an

American football team that is based here on Baltimore.

So, this the homepage of that team on ESPN.

Which is a sports channel in the U.S. and so

what we're going to do is actually look at the source code.

So, if you right click on the page and say view

source what you'll see is a source document that looks like this.

It's actually quite complicated.

That's the source code that actually

demonstrates, that's process to show you the

website that you actually see when you go navigate your browser to that website.

So we're going to actually drill into this website

source code and see if we can extract some information.

So again I'm going to pass the file URL.

This is the URL of that website, if you go back to the previous page.

So now, since I'm parsing an HTML file, instead of an XML

file, I'm going to use HTML tree parse instead of XML tree parse.

Um,the difference is our, it's different enough you want to

use HTML tree parse when you're parsing an HTML file.

And then I'm going to pass the command use internal equals true, so

that I can get all the different nodes inside of that, that file.

So now what I'm going to do is, I'm again going

to use the xpathSApply to programatically extract some components of this document.

So I'm going to start with the whole document again.

And now what I'm going to try to do is, I'm going to

again extract the XML value to the value inside of certain elements.

But I want to find very specific kinds of elements.

So here's what I'm going to do.

I'm going to look for elements.

That are list items li, and that have a particular class.

So they have class equal to score.

And so, what this is going to do is it's

going to go through the entire document, and any time it

sees a tag that is a, a list item,

it's going to check and see if it's class is score.

And if it's class is score it'll return the XML value.

So it turns out if you go back to this website and you look very very

carefully, That you can see for example there are these list items and

the class in this class is equal to the team name and so the

next element that I'm going to be extracting

on this page is actually the team name.

So it's the same thing.

I look for a list item with class equal

to team name and it will extract those team names.

So the way this works is you go to the website and

you find the tag and any attributes that you want to extract,

and then you need to write them into this xpath language to

extract only the data for those specific elements with xpath, as applied.

So, if I do this, I end up with the scores for each of the games and the teams.

So, I've scraped from that website information directly using

This is a very brief introduction to XML.

The xml packaging r and xml in general.

If you go and check out the xml tutorials for the

xml package there's a short one that's really good and a long

one that's extremely extensive and should only be read after reading the

short one so you get an idea of what is going on.

And then there's really outstanding guide to the XML

package that's linked to here as well that will

give you a lot of information about how you

can use XML to programatically extract information from websites.

Reading XML

Getting and Cleaning Data

与讲师见面