Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping. Three features make it powerful:

Valuable data that was once locked up in poorly-designed websites is now within your reach. Projects that would have taken hours take only minutes with Beautiful Soup.

Getting Started

1. Install BeautifulSoup

2. Create a script that uses BeautifulSoup

from BeautifulSoup import BeautifulSoup
import re
import urllib2

# download the page
response = urllib2.urlopen("http://www.boattrader.com/search-results/Type-any/Make-mako/Length-17,25/Zip-02445/Radius-100/Sort-Length:DESC/ ")
html = response.read()

# create a beautiful soup object
soup = BeautifulSoup(html)

# all links to detailed boat information have class lfloat
links = soup.findAll("a", { "class" : "lfloat" })
for link in links:
print link['href']
print link.string

# all prices are spans and have the class rfloat
prices = soup.findAll("span", { "class" : "rfloat" })
for price in prices:
print price
print price.string

# all boat images have attribute height=105
images = soup.findAll("img",height="105")
for image in images:
print image			# print the whole image tag
print image['src']	# print the url of the image only

Helpful resources:

• How to use python and Beautiful Soup to screen scrape the links from a Google Blog search
• BeautifulSoup Documentation

These instructions work on OS X and Linux. Putty also has tunneling capabilities, but those are beyond the scope of this post.

There are many uses for the technology.

SSH Tunneling Applications

1. Get online at a client site - If you’re visiting a client site and don’t have proxy access you can use a SSH SOCKS proxy to bypass the LAN proxy and get on the web
2. Encrypt traffic - to keep LAN admins from seeing what you’re looking at
3. Bypass corporate web filters - bypass corporate firewall to get access to Gmail, Facebook and other restricted resources.
4. Defeat QOS limitations - Bypass rate limiting and other service caps imposed by QOS rules
5. Access your database server - Access your database server even though remote access is disabled.

Access your database server

Port forward Postgresql


ssh -f root@server.yourdomain.com -L 5432:localhost:5432 -N


Access the web using a Socks Proxy

$ssh -D 9999 username@ip-address-of-ssh-server

Then go to Firefox->Preferences->Advanced->Network->Settings to configure the socksproxy. The address is localhost.

Further Reading on SSH Tunneling

• Kees Cook tells us how to tunnel DNS lookups
• Don McArthur points out his excellent article that addresses the same issue
• verevi says the FoxyProxy extension

I know that Google Docs is very convenient, but make sure not to store sensitive data on the Google Docs system.

You can read up on Google’s security at the link below.  I feel their document is a little misleading as it doesn’t fully disclose the face that information is not encrypted while in transit.

http://www.google.com/apps/intl/en/business/infrastructure_security.html

1.) You want to see if the DNS change you requested in live on the primary nameserver

hopper-macbook:~ andrewhopper$ nslookup
> server dns1.noris.net
Default server: dns1.noris.net
Address: 213.95.0.65#53
> set type=a
> puma.com
Server: dns1.noris.net
Address: 213.95.0.65#53


Name: puma.com
Address: 128.167.119.25

2.) You just configured Google apps for your domain and want to verify MX records have been properly configured.

hopper-macbook:~ andrewhopper$ nslookup
> set q=mx
> hopper.biz

Server: 208.59.247.45
Address: 208.59.247.45#53

Non-authoritative answer:
hopper.biz mail exchanger = 20 alt1.aspmx.l.google.com.
hopper.biz mail exchanger = 20 alt2.aspmx.l.google.com.
hopper.biz mail exchanger = 30 aspmx2.googlemail.com.
hopper.biz mail exchanger = 30 aspmx4.googlemail.com.
hopper.biz mail exchanger = 30 aspmx3.googlemail.com.
hopper.biz mail exchanger = 30 aspmx5.googlemail.com.
hopper.biz mail exchanger = 10 aspmx.l.google.com.

Authoritative answers can be found from:
hopper.biz nameserver = NS2.SLICEHOST.NET.
hopper.biz nameserver = NS3.SLICEHOST.NET.
hopper.biz nameserver = NS1.SLICEHOST.NET.
aspmx.l.google.com internet address = 209.85.221.60
alt1.aspmx.l.google.com internet address = 72.14.247.27
alt2.aspmx.l.google.com internet address = 209.85.222.78
aspmx2.googlemail.com internet address = 209.85.135.27
NS1.SLICEHOST.NET internet address = 67.23.4.57

The process will void your warranty but is pretty straight forward thanks to the phenomenal instructions here-iFixIt Macbook Pro 15 Hard Drive Replacement Guide

UPDATE employees SET sales_count = sales_count + 1 FROM accounts
  WHERE accounts.name = 'Acme Corporation'
  AND employees.id = accounts.sales_person;

Here’s a link to the man page - http://man.cx/screen

Here’s a quick cheatsheet borrowed from http://www.pixelbeat.org/lkdb/screen.html

VIM 5 Min Tutorial

"Here is your 5 minute tutorial. The easiest way to learn vi is to know what the letters stand for:"
y(ank) - copy
d(elete) - delete
c(hange) - change
p(aste) - put from buffer after cursor
o(pen) - start a new line
i(nsert) - insert before current character
a(fter) - insert after current character
w(ord) - moves to beginning of next word
b(ack) - moves to beginning of current word or prior word
e(end) - moves to end of current word or next word
f(ind) - moves to a character on the current line
movement keys you just need to learn: h,j,k,l

^ - beginning of text on a line
$ - end of text on a line
0 - first position on line

most commands can be prefaced with numeric modifiers.
2w - means move 2 words
5h - means move 5 charcters to the left
3k - means move 3 lines up
3fs - means move to the 3rd letter s folling the cursor

modification commands (d,c,y) need to know how much to work on.
dd - delete a line into memory
yy - yank a line into memory
cc - change the whole line
c$ - change from current position to the end
c2w - change the text spanning the next 2 words
3dd - delete 3 lines
d2f. - delete to the second period.

. - means redo the last modification command.
/ - searches for text, and then n(ext) will go the next found occurance. N will go prior.
? - searches backwards through the document.

Further Reading about VIM

1. http://rayninfo.co.uk/vimtips.html
2. http://blog.interlinked.org/tutorials/vim_tutorial.html
3. http://www.viemu.com/a_vi_vim_graphical_cheat_sheet_tutorial.html