"Hacking" a website?

[tadka]

I’m faces with a tedious task of getting a big list of data on BMW cars. I have to go to a parts website and check which parts are compatible with which model variations. And there are a lot of model variants. Is there a way to simplify this? I’m thinking maybe some program that can access “raw” data on the website? Or is it all scattered through the databases and there is no easy way?

[Ed P]

There is but you need to familiarise yourself with Python and beautiful soup, feedparser and maybe pandas.

It is not hard as there are plenty of examples on line, but it is not trivial and some web sites object to being ‘scraped’.

Start with the beautiful soup site and possibly search on line for ‘scraping’. Someone may have done something similar to your task and put the code on-line for your use. 99% of such code will use Python.

e.g. search for (say) web scraping+car data

 

[tadka]

Ed P wrote:

There is but you need to familiarise yourself with Python and beautiful soup, feedparser and maybe pandas. It is not hard as there are plenty of examples on line, but it is not trivial and some web sites object to being ‘scraped’. Start with the beautiful soup site and possibly search on line for ‘scraping’. Someone may have done something similar to your task and put the code on-line for your use. 99% of such code will use Python. e.g. search for (say) web scraping+car data

Thanks Ed P. Tho it looks even more complicated than what I am doing. Especially now that I got into the swing of it. I’ll see how it goes.

[Ed P]

If you want to try a bit of Python the following  script MAY help:

# code follows,

#!/usr/bin/python2.7
# -*- coding: utf-8 -*-

#the above line tells the script is uses python 2.7, you will have to look up how to import beautiful soup and urllib2 for your os

# the next just sets the fonts used

# apologies but  I cannot remember from where this was cribbed

 

from BeautifulSoup import BeautifulSoup

import urllib2
url = urllib2.urlopen(“https://address of page you want to scrape”)

content = url.read()

soup = BeautifulSoup(content)

print soup.prettify()

# code snippet ends

[tadka]

Ed P wrote:

If you want to try a bit of Python the following script MAY help: # code follows, #!/usr/bin/python2.7 # -*- coding: utf-8 -*- #the above line tells the script is uses python 2.7, you will have to look up how to import beautiful soup and urllib2 for your os # the next just sets the fonts used # apologies but I cannot remember from where this was cribbed from BeautifulSoup import BeautifulSoup import urllib2 url = urllib2.urlopen(“https://address of page you want to scrape”) content = url.read() soup = BeautifulSoup(content) print soup.prettify() # code snippet ends

You lost me at “the following scrip”  😛

I spent the entire day searching and cross-referencing. My brain is done…

[Mark Turner]

At work we have an old copy of the porsche PET system you can either put in a part number directly and it will tell you what models it was used on and if the part has a superseded part number or you could pick which model then go through all the sub menus and pictures till you find the part you need then it will tell you the part number and what vehicles it was used on. I’m sure BMW would have a similar system. I’m sure there would be a version of said software on the internet or you could ask a friendly BMW specialist nicely if they could help. Its worth a try.

 

[tadka]

Mark Turner wrote:

At work we have an old copy of the porsche PET system you can either put in a part number directly and it will tell you what models it was used on and if the part has a superseded part number or you could pick which model then go through all the sub menus and pictures till you find the part you need then it will tell you the part number and what vehicles it was used on. I’m sure BMW would have a similar system. I’m sure there would be a version of said software on the internet or you could ask a friendly BMW specialist nicely if they could help. Its worth a try.

There is a website like that, it’s called Realoem. But they don’t list the specs of the special tools needed for those parts. I was making a list of cars for each tool. Didn’t get it all done but good enough for eBay I suppose.

[Ed P]

Sorry I confused you!

Python has a huge number of routines like beautiful soup which are really general purpose programs, very powerful but their names often have no real world meaning. That short script takes the html code for a site and tries to turn it into a form in which data can be more readily accessed.

[Author]

Posts