Published: 2011-04-29
A command-line script (written in Python) for validating HTML and CSS files and URLs using the WC3 validators.
The w3c-validator.py script uses the curl(1) command to submit HTML files and URLs to the W3C Markup Validation Service and CSS files and URLs to the W3C CSS Validation Service. The script parses and reports the JSON results returned by the validators.
|
Note |
Currently the CSS validator’s JSON output option is experimental and not formally documented. |
You can find the source on GitHub at https://github.com/srackham/w3c-validator
The script command syntax is:
python w3c-validator.py [--verbose] FILE|URL...
Examples (w3c-validator is just a convenient symbolic link in the shell PATH to the executable w3c-validator.py script):
$ w3c-validator tests/data/testcases-html5.html validating: tests/data/testcases-html5.html ... error: line 822: The tt element is obsolete. Use CSS instead. error: line 829: The tt element is obsolete. Use CSS instead. error: line 838: The tt element is obsolete. Use CSS instead. |
$ w3c-validator http://www.methods.co.nz/asciidoc/layout1.css validating: http://www.methods.co.nz/asciidoc/layout1.css ... |
Written and tested on Xubuntu 10.04 with Python 2.6.5, should work on other Python platforms.
]]>Published: 2011-03-18
A didactic Web application written in Python to illustrate how to use the Bottle web-framework with the MongoDB database.
It’s a port I made of Mike Dirolf’s DjanMon application (how to use Django with MongoDB).
Bottle is a wonderful micro web-framework, it’s beautifully written and documented and has that “just right” feel about it. Bottle has support for multiple webservers and template engines — I’m really enjoying it after working with Django.
MongoDB is a schemaless document-oriented database designed for the Web. For me, working with MongoDB has been a revelation. Database creation, administration and migration (long the bane of database developers) is trivial in comparison with traditional SQL databases. Document storage management is baked into MongoDB and is used in many web applications (document storage has always been awkward and slow with relational databases).
You can find the source on GitHub.
Here’s a screenshot:
To run the application install the prerequisite packages then:
git clone https://github.com/srackham/bottle-mongodb-example.git cd bottle-mongodb-example python bottle-mongodb-example.py
The application creates and displays a paged list of user created messages which are stored in a MongoDB database along with uploaded images (no file system management required it’s all in the database) — not bad for 93 lines of Python code and a 79 line HTML template. The really neat thing is that you don’t have to create the database, tables or schema — it all happens automatically.
sudo easy_install pymongo
git clone https://github.com/defnull/bottle.git cd bottle python setup.py install
I’m running in this environment — it works for me but your mileage may vary:
Published: 2010-06-19
Updated: 2010-06-20
Updated: 2013-12-08
Using the VirtualBox GUI to manually merge lots of snapshots is time consuming and fiddly so I wrote a Python script called vboxmerge.py to do this automatically. The script merges a snapshot branch into the base VDI with a single command (it also illustrates how easy it is to script VirtualBox using Python).
|
Important |
This post is obsolete: a much easier way is to use the VirtualBox VM Clone command — see Update December 2013: An easier way to merge snapshots in my Cloning and Copying VirtualBox virtual machines blog post. |
The more VirtualBox snapshots you have the more disk space you consume. Snapshot VDIs grow with every previously unallocated guest OS write and it’s not long before the total size of a machine’s base VDI plus snapshots exceeds than the machines logical HDD size (this is a very good reason why it’s not a good idea to oversize your hard disks).
Here’s an example of the savings on one of my Windows XP guests (it has a 50GB logical HDD):
The merge almost halved the size; the compaction brought it to to below 10% of the original total size.
Compacting without first zeroing out free space on the guest file system generally provides no or very little benefit.
If you’re unfamiliar with the workings of VirtualBox snapshots and VDIs there is a great set of FAQs and Tutorial: All about VDIs in the VBox forums.
--skip=N and --count=N options limit the merge to a snapshot subset. --compact option compacts the machine’s base VDI. --snapshot option creates a base snapshot following the merge. --discard-currentstate option discards the machine’s current state before merging. python vboxmerge.py --help to display command options. To merge snapshots into the base VDI just run the script specifying the machine name e.g.
python vboxmerge.py "My Machine"
If the machine’s snapshot tree has multiple branches you will need to run the vboxmerge once for each branch to merge the entire tree. The final machine state will the that of the last merged branch.
|
Important |
Backup all your virtual machines before running the script and do dry runs first (using the --dryrun command option). |
|
Note |
|
Gotcha
If the machine being processed is selected in the VirtualBox GUI then the GUI sometimes throws and error or stops responding. No damage results and the problem can be avoided by selecting another machine in the GUI before you run vboxmerge.
Merging snapshots removes redundant shadowed blocks and will return a lot of space, but it doesn’t return blocks that have previously been written but are now no longer used by the guest file system. VirtualBox processes VDIs as block devices, it knows nothing about files and file systems. The VirtualBox compact command compacts blocks of zeros so for compaction to be effective you need to zero free space from the guest operating system before compacting the VDI using VirtualBox.
| Windows |
Use sdelete, for example to zero all unused space on drive C: sdelete.exe -c C: |
| Linux |
A zeroing utility for ext2 and ext3 file systems is zerofree written by Ron Yorston. Your file system must be unmounted or mounted read-only before using this utility. |
If you decide to compact you should zero out the free space after merging (if you do it before merging the zeroing will balloon the most recent snapshot and the subsequent merging will take much longer). The steps are:
python vboxmerge.py "My Machine"
python vboxmerge.py --compact --snapshot "My Machine"
It would be nice if file systems had some sort of zero after delete option to zero free disk space automatically (see zerofree).
Why create a base snapshot?
Because I never like to write directly to the base VDI — creating a snapshot effectively leaves the base VDI read-only so even if your machine crashes half way through writing and corrupts the current state you can restore back to the base VDI.
This is the environment I used to develop and test vboxmerge.py:
You will also need to download and install Python for Windows extensions (in my case 64bit).
Setup the Python VirtualBox bindings wrapper module (vboxapi) which is installed by the VirtualBox installer:
C:\> cd C:\Program Files\Oracle\VirtualBox\sdk\install C:\> C:\Python26\python.exe vboxapisetup.py install |
Test your environment by running Python and executing:
>>> import vboxapi >>> vb = vboxapi.VirtualBoxManager(None,None) |
If there a no errors you’re OK.
To print a list of your VM names and UUIDs execute this:
>>> for m in vb.getArray(vb.vbox,'machines'): ... print m.name, m.id ... |
|
Important |
The script was developed and tested with VirtualBox 3.2.4, it might not work with other versions of VirtualBox (see Prerequisites). |
#!/usr/bin/env python ''' vboxmerge.py - Merge VirtualBox snapshots into base VDI Run 'python vboxmerge.py --help' to display command options. Written by Stuart Rackham, <[email protected]> Copyright (C) 2010 Stuart Rackham. Free use of this software is granted under the terms of the MIT License. ''' import os, sys import vboxapi import pywintypes PROG = os.path.basename(__file__) VERSION = '0.1.1' VBOX = vboxapi.VirtualBoxReflectionInfo(False) # VirtualBox constants. def out(fmt, *args): if not OPTIONS.quiet: sys.stdout.write((fmt % args)) def die(msg, exitcode=1): OPTIONS.quiet = False out('ERROR: %s\n', msg) sys.exit(exitcode) def runcmd(async_cmd, *args): ''' Run the bound asynchronous method async_cmd with arguments args. Display progress and return once the command has completed. If an error occurs print the error and exit the program. ''' if not OPTIONS.dryrun: try: progress = async_cmd(*args) while not progress.completed: progress.waitForCompletion(30000) # Update progress every 30 seconds. out('%s%% ', progress.percent) out('\n') except pywintypes.com_error, e: die(e.args[2][2]) # Print COM error textual description and exit. def vboxmerge(machine_name): ''' Merge snapshots using global OPTIONS. ''' vbm = vboxapi.VirtualBoxManager(None, None) vbox = vbm.vbox try: mach = vbox.findMachine(machine_name) except pywintypes.com_error: die('machine not found: %s' % machine_name) out('\nmachine: %s: %s\n', mach.name, mach.id) if mach.state != VBOX.MachineState_PoweredOff: die('machine must be powered off') session = vbm.mgr.getSessionObject(vbox) vbox.openSession(session, mach.id) try: snap = mach.currentSnapshot if snap: if OPTIONS.discard_currentstate: out('\ndiscarding current machine state\n') runcmd(session.console.restoreSnapshot, snap) skip = int(OPTIONS.skip) count = int(OPTIONS.count) while snap: parent = snap.parent if skip <= 0 and count > 0: out('\nmerging: %s: %s\n', snap.name, snap.id) runcmd(session.console.deleteSnapshot, snap.id) # The deleteSnapshot API sometimes silently skips snapshots # so test to make sure the snapshot is no longer valid. try: snap.id except pywintypes.com_error: pass else: if not OPTIONS.dryrun: die('%s: %s: more than one child VDI' % (snap.name, snap.id)) count -= 1 snap = parent skip -= 1 else: out('no snapshots\n') if OPTIONS.snapshot: # Create a base snapshot. out('\ncreating base snapshot\n') runcmd(session.console.takeSnapshot, 'Base', 'Created by vboxmerge') if OPTIONS.compact: # Compact the base VDI. for attachment in mach.mediumAttachments: if attachment.type == VBOX.DeviceType_HardDisk: base = attachment.medium.base if base.type == VBOX.MediumType_Normal: out('\ncompacting base VDI: %s\n', base.name) runcmd(base.compact) finally: session.close() if __name__ == '__main__': description = '''Merge VirtualBox snapshots into base VDI. MACHINE is the machine name.''' from optparse import OptionParser parser = OptionParser(usage='%prog [OPTIONS] MACHINE', version='%s %s' % (PROG,VERSION), description=description) parser.add_option('--skip', dest='skip', help='skip most recent N snapshots', metavar='N', default=0) parser.add_option('--count', dest='count', help='only merge N snapshots', metavar='N', default=1000) parser.add_option('-q', '--quiet', action='store_true', dest='quiet', default=False, help='do not display progress messages') parser.add_option('-n', '--dryrun', action='store_true', dest='dryrun', default=False, help='do nothing except display what would be done') parser.add_option( '--compact', action='store_true', dest='compact', default=False, help='compact the base VDI') parser.add_option( '--snapshot', action='store_true', dest='snapshot', default=False, help='create base snapshot') parser.add_option( '--discard-currentstate', action='store_true', dest='discard_currentstate', default=False, help='discard the current state of the MACHINE') if len(sys.argv) == 1: parser.parse_args(['--help']) global OPTIONS (OPTIONS, args) = parser.parse_args() vboxmerge(args[0]) |
C:\Program Files\Oracle\VirtualBox\sdk\install\vboxapi\VirtualBox_constants.py. Published: 2010-03-23
Updated: 2010-03-30, 2010-05-01, 2011-06-10
|
Note |
The current version of this app is written in CoffeeScript and is published in the npm package registry. |
A simple Python command-line script I wrote to send SMS messages using Clickatell’s HTTP API. In addition it:
-p command option).
$ python sms.py 64912345667 "Hello World"
ID: 26a8147fa04ed9fj2a9ad125c55cee00
$ sms.py -s 26a8147fa04ed9fj2a9ad125c55cee00
apiMsgId: 26a8147fa04ed9fj2a9ad125c55cee00 charge: 0.8 status: 004
(received by recipient)
$ python sms.py -b
Credit: 204.900
$ python sms.py --help
NAME
sms.py - Send SMS message
SYNOPSIS
sms.py PHONE MESSAGE
sms.py -s MSGID
sms.py -b | -l
DESCRIPTION
A simple Python command-line script to send SMS messages using
Clickatell's HTTP API (see http://clickatell.com).
Records messages log in /home/srackham/sms.log
Reads configuration parameters from /home/srackham/.sms.conf
OPTIONS
-s MSGID
Query message delivery status.
-b
Query account balance.
-l
List message log file using /usr/bin/less.
-p
List phone book.
AUTHOR
Written by Stuart Rackham,
COPYING
Copyright (C) 2010 Stuart Rackham. Free use of this software is
granted under the terms of the MIT License.
|
Set the Clickatell account configuration parameters in the sms.py script or put them in a JSON format file named .sms.conf in your home directory. For example:
{
"USERNAME": "foobar",
"PASSWORD": "secret",
"API_ID": "123456",
"SENDER_ID": "+64912345678",
"PHONE_BOOK": {
"tom": "+64 21 1234 5678",
"dick": "+61 25 1234 567",
"harry": "+64 9 1234 346"
}
}
|
#!/usr/bin/env python
# A simple Python command-line script to send SMS messages using
# Clickatell's HTTP API (see http://clickatell.com).
'''
NAME
%(prog)s - Send SMS message
SYNOPSIS
%(prog)s PHONE MESSAGE
%(prog)s -s MSGID
%(prog)s -b | -l
DESCRIPTION
A simple Python command-line script to send SMS messages using
Clickatell's HTTP API (see http://clickatell.com).
Records messages log in %(log)s
Reads configuration parameters from %(conf)s
OPTIONS
-s MSGID
Query message delivery status.
-b
Query account balance.
-l
List message log file using %(pager)s.
-p
List phone book.
AUTHOR
Written by Stuart Rackham, <[email protected]>
COPYING
Copyright (C) 2010 Stuart Rackham. Free use of this software is
granted under the terms of the MIT License.
'''
VERSION = '0.2.4'
import urllib
import os, sys, time, re
import simplejson as json
# Clickatell account configuration parameters.
# Create a separate configuration file named `.sms.conf` in your `$HOME`
# directory. The configuration file is single JSON formatted object with the
# same attributes and attribute types as the default CONF variable below.
# Alternatively you could dispense with the configuration file and edit the
# values in the CONF variable below.
CONF = {
'USERNAME': '',
'PASSWORD': '',
'API_ID': '',
'SENDER_ID': '', # Your registered mobile phone number.
'PHONE_BOOK': {},
}
PROG = os.path.basename(__file__)
HOME = os.environ.get('HOME', os.environ.get('HOMEPATH',
os.path.dirname(os.path.realpath(__file__))))
LOG_FILE = os.path.join(HOME,'sms.log')
CONF_FILE = os.path.join(HOME,'.sms.conf')
if os.path.isfile(CONF_FILE):
CONF = json.load(open(CONF_FILE))
SENDER_ID = CONF['SENDER_ID']
PHONE_BOOK = CONF['PHONE_BOOK']
PAGER = os.environ.get('PAGER','less')
# URL query string parameters.
QUERY = {'user': CONF['USERNAME'], 'password': CONF['PASSWORD'],
'api_id': CONF['API_ID'], 'concat': 3}
# Clickatell status messages.
MSG_STATUS = {
'001': 'message unknown',
'002': 'message queued',
'003': 'delivered to gateway',
'004': 'received by recipient',
'005': 'error with message',
'007': 'error delivering message',
'008': 'OK',
'009': 'routing error',
'012': 'out of credit',
}
# Retrieve Clickatell account balance.
def account_balance():
return http_cmd('getbalance')
# Query the status of a previously sent message.
def message_status(msgid):
QUERY['apimsgid'] = msgid
result = http_cmd('getmsgcharge')
result += ' (' + MSG_STATUS.get(result[-3:],'') + ')'
return result
# Strip number punctuation and check the number is not obviously illegal.
def sanitize_phone_number(number):
result = re.sub(r'[+ ()-]', '', number)
if not re.match(r'^\d+$', result):
print 'illegal phone number: %s' % number
exit(1)
return result
# Send text message. The recipient phone number can be a phone number
# or the name of a phone book entry.
def send_message(text, to):
if to in PHONE_BOOK:
name = to
to = PHONE_BOOK[to]
else:
name = None
to = sanitize_phone_number(to)
sender_id = sanitize_phone_number(SENDER_ID)
if sender_id.startswith('64') and to.startswith('6427'):
# Use local number format if sending to Telecom NZ mobile from a NZ
# number (to work around Telecom NZ blocking NZ originating messages
# from Clickatell).
sender_id = '0' + sender_id[2:]
QUERY['from'] = sender_id
QUERY['to'] = to
QUERY['text'] = text
result = http_cmd('sendmsg')
now = time.localtime(time.time())
if name:
to += ': ' + name
print >>open(LOG_FILE, 'a'), \
'to: %s\nfrom: %s\ndate: %s\nresult: %s\ntext: %s\n' % \
(to, sender_id, time.asctime(now), result, text)
return result
# Execute Clickatell HTTP command.
def http_cmd(cmd):
url = 'http://api.clickatell.com/http/' + cmd
query = urllib.urlencode(QUERY)
file = urllib.urlopen(url, query)
result = file.read()
file.close()
return result
if __name__ == '__main__':
argc = len(sys.argv)
if argc == 3:
if sys.argv[1] == '-s':
print message_status(sys.argv[2])
else:
print send_message(sys.argv[2], sys.argv[1])
elif argc == 2 and sys.argv[1] == '-b':
print account_balance()
elif argc == 2 and sys.argv[1] == '-l': # View log file.
os.system(PAGER + ' ' + LOG_FILE)
elif argc == 2 and sys.argv[1] == '-p': # List phone book.
for i in PHONE_BOOK.items():
print '%s: %s' % i
else:
print __doc__ % {'prog':PROG,'log':LOG_FILE,'conf':CONF_FILE,'pager':PAGER}
|