If you’re not already familiar with the concept I bet you’d be very interested to learn that you can simulate VPN access to a network as long as you have access to a linux host on that network.
These instructions work on OS X and Linux. Putty also has tunneling capabilities, but those are beyond the scope of this post.
There are many uses for the technology.
SSH Tunneling Applications
- Get online at a client site - If you’re visiting a client site and don’t have proxy access you can use a SSH SOCKS proxy to bypass the LAN proxy and get on the web
- Encrypt traffic - to keep LAN admins from seeing what you’re looking at
- Bypass corporate web filters - bypass corporate firewall to get access to Gmail, Facebook and other restricted resources.
- Defeat QOS limitations - Bypass rate limiting and other service caps imposed by QOS rules
- Access your database server - Access your database server even though remote access is disabled.
Access your database server
Port forward Postgresql
ssh -f root@server.yourdomain.com -L 5432:localhost:5432 -N
Access the web using a Socks Proxy
$ssh -D 9999 username@ip-address-of-ssh-server
Then go to Firefox->Preferences->Advanced->Network->Settings to configure the socksproxy. The address is localhost.
Further Reading on SSH Tunneling
- Kees Cook tells us how to tunnel DNS lookups
- Don McArthur points out his excellent article that addresses the same issue
- verevi says the FoxyProxy extension
BeautifulSoup Demo - screen scraping quickly with Python
From the BeautifulSoup website:
Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping. Three features make it powerful:
1. Beautiful Soup won’t choke if you give it bad markup. It yields a parse tree that makes approximately as much sense as your original document. This is usually good enough to collect the data you need and run away.
2. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. You don’t have to create a custom parser for each application.
3. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don’t have to think about encodings, unless the document doesn’t specify an encoding and Beautiful Soup can’t autodetect one. Then you just have to specify the original encoding.
Beautiful Soup parses anything you give it, and does the tree traversal stuff for you. You can tell it “Find all the links”, or “Find all the links of class externalLink”, or “Find all the links whose urls match “foo.com”, or “Find the table heading that’s got bold text, then give me that text.”
Valuable data that was once locked up in poorly-designed websites is now within your reach. Projects that would have taken hours take only minutes with Beautiful Soup.
Getting Started
1. Install BeautifulSoup
2. Create a script that uses BeautifulSoup
from BeautifulSoup import BeautifulSoup import re import urllib2 # download the page response = urllib2.urlopen("http://www.boattrader.com/search-results/Type-any/Make-mako/Length-17,25/Zip-02445/Radius-100/Sort-Length:DESC/ ") html = response.read() # create a beautiful soup object soup = BeautifulSoup(html) # all links to detailed boat information have class lfloat links = soup.findAll("a", { "class" : "lfloat" }) for link in links: print link['href'] print link.string # all prices are spans and have the class rfloat prices = soup.findAll("span", { "class" : "rfloat" }) for price in prices: print price print price.string # all boat images have attribute height=105 images = soup.findAll("img",height="105") for image in images: print image # print the whole image tag print image['src'] # print the url of the image onlyHelpful resources: