Swagger Schema

First, our database platform (InterSystems Iris) provided a way to get a Swagger spec (link) which pulls information from a target dispatch class. For the sake of this post, this is the relevant schema I was after:

{
  "paths": {
    "/some/url": {
      "post": {
        "description": "Extracted from ClassMethod Comments.",
        "summary": "Extracted from route comments.",
      }
    } 
  },
  "swagger":"2.0"
}

Now to loop over the swagger spec and get a list of records describing each API, we loop over a dictionary of dictionaries:

# Pass in the entire swagger data object
def processSwaggerRecord(jsonData):
  swaggerSpecs = []
  for path, spec in jsonData['paths'].items():
    for pathType, apiData in spec.items():
      record = {}
      record['url'] = path
      record['type'] = pathType.upper()
      record['Total'] = 0
      swaggerSpecs.append(record)
  return swaggerSpecs

Postman Schema

Next, we had several postman collections defined to test various aspects of our APIs. To do a rough gap analysis, I needed to extract the routes and request types so I could match each request against the swagger spec and create a tally. Here is the relevant schema in the exported postman collections:

{
  "info": {
    "_postman_id": "...",
    "name": "My Postman Collection",
    "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
  },
  "item": [{
    "name": "Testing - Non-existant user",
    "request": {
      "auth": {"type": "oauth2"},
      "method": "GET",
      "url": {
        "raw": "{{url}}/some/url",
        "host": ["{{url}}"],
        "path": ["some","url"],
        "query": [{"key": "foo", "value": "bar"}]
      }
    },
    "response": []
  }],
}

Postman allows you to group requests in folders to an arbitrary depth. Both request and folder objects appear as records within the item property. Therefore, we can tell them apart based on the existence of either request/response properties (indicating a request) or an item property (indicating a sub-folder). Taking a request object, we want to extract the same information that we extracted from the Swagger data:

# pass in an item array
def processPostmanFolder(postmanFolder):
  output = []
  for request in postmanFolder:
    if 'item' in request:
      output += processPostmanFolder(request['item'])
    else:
      output.append(extractPostmanRecord(request))
  return output


def extractPostmanRecord(postmanRequest):
  record = {}
  record['name'] = postmanRequest['name']
  record['type'] = postmanRequest['request']['method']
  rawUrl = postmanRequest['request']['url']['raw']
  queryIndex = rawUrl.find('?')
  record['queryString'] = ''
  if (queryIndex > -1):
    record['queryString'] = rawUrl[queryIndex+1:]
    rawUrl = rawUrl[0:queryIndex]
  record['url'] = rawUrl[rawUrl.find('/'):]
  return record

Matching Requests

Since postman requests can have values defined where the swagger spec indicates variable data, we cannot rely on exact route matches alone. So, if we cannot find an exact match on the route, then we can try matching on just the static parts of the route:

def requestsMatch(swaggerAPI, postmanRequest):
  if ( (postmanRequest['type'] != None) and (swaggerAPI['type'] != postmanRequest['type']) ):
    return False
  if (swaggerAPI['url'] == postmanRequest['url']):
    return True
  #
  # If the routes contain parameters and/or environment
  # variables, try to match on just the static part.
  swaggerIndex = swaggerAPI['url'].find("/{")
  if (swaggerIndex > -1):
    swaggerUrl = swaggerAPI['url'][0:swaggerIndex]
    postmanIndex = postmanRequest['url'].find("/{{")
    if (postmanIndex > -1):
      postmanUrl = postmanRequest['url'][0:postmanIndex]
      if (swaggerUrl == postmanUrl):
        return True
      else:
        postmanUrl = postmanRequest['url'][0:len(swaggerUrl)]
        if (swaggerUrl == postmanUrl):
          return True
  return False

The Final Script

Now that we have all the basic pieces, it is time to put them together. For the postman collections, I decided to drop all the exported collection files in a ./data folder and define a tab-delimited file collections.csv where I can specify an intermediate CSV file and a collection specific tally column header for each source collection file. (I prefer having my script save each processed postman collection to a csv as a way to manually verify the script was working as intended.) Then it compiles a tally of how many times each request appears in a collection specific column. Finally, it generates the gap analysis report csv.

def process(swaggerFile):
  print("Loading Swagger File.")
  swaggerSpecs = extractSwaggerData(swaggerFile)

  # Get the defined collection catalogue list
  postmanCollections = loadPostmanCatalogue()
  fieldNames = ['url', 'type', 'applications', 'permissions check', 'permissions']
  tallyColumns = []
  postmanFields = ['url', 'queryString', 'type', 'name']

  # Loop over all exported collection files
  for collection, file, column in postmanCollections:
    print("\nLoading " + collection)
    postmanRequests = extractPostmanCollection(os.path.join('.//data//', collection))

    # so we can check our work if necessary
    saveToCSV(file, postmanFields, postmanRequests)

    # update tallies for each postman collection
    swaggerSpecs = tallyPostmanRequests(swaggerSpecs, postmanRequests, column)
    fieldNames.append(column)

  fieldNames = fieldNames + ['Total',
                 'summary', 'description']
  saveToCSV('.//API_Catalogue.csv',
            fieldNames, swaggerSpecs)
  return


#
# Update the request tallies in apiList based on the
#  requests in postmanCollection
#
def tallyPostmanRequests(apiList, postmanCollection, tallyColumn):
  for request in postmanCollection:
    match = False
    for record in apiList:
      if tallyColumn not in record:
        record[tallyColumn] = 0
      if requestsMatch(record, request):
        record[tallyColumn] = record[tallyColumn] + 1
        record['Total'] = record['Total'] + 1
        match = True
        continue
      if (match == False):
        print("Could not match postman request for " + request['url'])
  return apiList

If you have collections broken out topically, then the resulting tally columns can give you a rough picture of what your coverage looks like. For example, we had one collection that explicitly checked our secure APIs to make sure requests missing the necessary authentication returned 403 Reponses. Another collection could verify all the “missing input required” validation conditions. Another collection could be all APIs not requiring auth. etc…

The full code can be found here on my github account.

Optional Parameters Are (Sometimes) Required

In php, all function parameters are required unless you give them a default value.

function foo($var1, $var2="something") {
  // do stuff
  return;
}

In this case, $var1 is required whereas $var2 is optional with a default value of empty string.  It would be perfectly acceptable to call it like this:

foo("bar");

What if we wanted to add another optional parameter?

function foo($var1, $var2="something", $var3="else") {
  // do stuff
  return;
}

PHP will not let you override $var3 without also providing a value for $var2 even though it has a default value. So, if you wanted to keep the default value for $var2, you would need to pass it in like so:

foo("bar", "something", "different");

One or two arguments are not a problem. Three or more and this gets cumbersome. That is why it is common practice to bundle together the optional arguments into an array and handling the defaults within the function rather than the function definition.

function foo($var1, $var2="something", $var3="else") {
  // do stuff
  return;
}

foo("bar", Array("var2"=>"something", "var3"=>"else");

** However, as I’m writing up this post, I discovered that there is a way to pass a variable number of arguments using func-get-args, func-get-arg, and func-num-args which I didn’t notice at the time I encountered this problem.

For Loops

In addition to C style syntax, php also provides a foreach statement which can be used in two forms:

// list of values or objects
foreach ( $array as $obj ) { ... }

// associative list
foreach ( $array as $key=>$value ) { ... }

This is the inverse of Python’s syntax which is:

for element in list:
  # ...

for (x, y) in list_of_tuples:
  # ...

While learning Python’s syntax first may play a part in my bias, I think I still favor it because it mirrors set theory.  However that is more personal preference than anything else.

Substrings

To extract a substring, you need this command:

substr( $string, $start, $length)

I kept putting the end position in there for the longest time.  Here, I really miss python’s array slicing:

string[start, end] 

Troubleshooting

I don’t know how many times the magic of var_dump() has helped me arrive at a solution in 30 seconds vs. 30+ min of searching for my answer in documentation or a blog post somewhere.  var_dump() is your friend.  It is especially enlightening when troubleshooting objects you are unfamiliar with, which was often for me as I was learning WordPress.

Presentations I’m Interested In

To date, Django is the only Python framework I’ve had the pleasure of working with.  I’m curious to see what Kennith Reitz‘s has to say in his presentation Flasky Goodness because it never hurts to diversify your toolbox.  The other talk that looks fascinating on Saturday is APIs for Mobile Networks(Chris Ridenour).  I’m curious to hear about the challenges and solutions the presenters encountered when dropping from wifi into mobile network land.  Finally, the workshop by David Stanek and Mike Pirnat on Web Security & Django with its hands on approach to common security vulnerabilities looks like it has tons of value.

Another mobile centric talk I’m excited to see is Django Powered Mobile Apps by Tim Kuelhorn.  Additionally, I’d like to see My Adventures with Mock (Brad Montgomery) to take my testing skills to the next level.  Finally, I want to hit Python + Fuse (Zach Wick).

Sprinting

As of this writing, the only planned sprint is on Python Core.  I participated in this last year and had a great time.  However, I waited until the night of the sprint to get set up with everything I needed which seriously detracted from me being able to fully participate.  (Figuring out and installing dependencies, then downloading & compiling the code all over a very slow wifi connection… you get the idea).

If you plan to participate, I cannot stress and recommend enough the importance of getting set up beforehand.  The Python Developer’s Guide has a quick start section for getting everything you need.

sudo apt-get install mercurial
hg clone http://hg.python.org/cpython

sudo pip install Django --upgrade

Verify from python shell:

import django
print django.get_version()

Thank you stackoverflow.

Django bootcamp

The best thing I got out of this presentation by Raymond Chandler was setting up a basic development environment on Ubuntu and getting comfortable with git and virtualenv in a Linux environment.  Then we progressed to build a basic twitter clone called ‘chirp’.

setting up a development environment:

sudo apt-get install git
pip install django
pip install virtualenv

virtualenv

activate/deactivate
virtualenv [name]
source [name]/bin/activate
pip install django  (installs within the environment)

setting up app

As part of the process of setting up a Django App, Raymond suggested renaming the project app directory to “config” (a rails convention), which I kind of like:

django-admin.py starproject chirp
mv chirp/ config ## rails convention

• edit manage.py to change os.environ.(…) settings
• edit chirp/config/settings.py to change [app name] to config

GitHub

Getting a new git repository started for the demo app “chirp”…

python2 manage.py startapp chirp
git init
touch .gitignore
[edit .gitignore]
git add .gitignore
git status
git add --all
git commit -m "commit message"

git remote add origin [location]
git push -u origin master
-u: makes repo you are pushing to the athoritative copy

git checkout -b adding-chirp-model

Miscellaneous

• django command extensions
• tastypy :: restful web services app
• python manage.py inspectdb :: assimilates database, creates model
• vim plugins: nerdtree, miniBufEx
• help([module]plorer)

Presentation notes can be found here: http://www.scribd.com/doc/101340711/Django-Bootcamp

Testing

http://drunkenpython.org/

Assertion

assert 1==2, “Reality Check”

python -i myModule.py #provides interactive debugger
python -o #switch to deactivate assertions
python -m unittest [module[.TestClass[.assertion]]] #switch to load module

unittest module

import unittest
dir(unittest)   # list out methods
subclass unittest.TestCase
– you add methods called test
– supply assert* methods

unittest output:

• (period): successful
• F: Failed
• E: Error in assertion

outcomes accumulated in unittest.TestResult instance

can create multiple subclasses and create testsuites
str_suite = unittest.TestSuite()
str_suite.addTest(class(“Descriptor”))

Test fixture: tests + suites + Setup + teardown

pip install distribute

doctest

python -m doctest pycampNG/fibonacci.py

– inspects file for doc strings and constructs and attempts to execute code

doctest.testmod() # runs all test in specified module
doctest.DocTestSuite() converts doctests to a unittest.TestSuite instance.
– reference in setup.py

Lightning talks 1

• Midwest python workshop for women
  • @catherinedevlin
• Hack day
  • @mpirnat
  • github.com/mpirnat/hackday
• gittip: https://www.gittip.com/
• zoopshop
  • https://zoopshop.com/
  • Built with responsive HTML5
  • Only about 1/300 ppl click through Facebook store.
  • Idea: by items directly from within FB news feed.
• Hieroglyph
  • take Sphinx/restructured docs and turn them into presentation
  • https://github.com/nyergler/hieroglyph
• python + weather
  • weather IRC channel
  • github.com/jennybot
  • “faster than NOAA, weather.com, nbc”

Brandon Rhodes: Python Patterns 1

1977 Architect Christopher Alexander
– Pattern at one level of detail, apply it to any scale.
“Gang of Four”: Gamma, Helm, Johnson, Vlissides :: Elements of reusable Object design

Not all patterns are OO patterns:

•  dictionary has a get(key, default) method.
•  continue: break out an iteration of a for loop, continue on to the next iteration

– Python’s dynamic features makes OO patterns disappear

builder design patterns

• python doesn’t have separate syntax for instantiating objects
• most “creational oo patterns” are contortions to avoid making brittle ‘new’ calls in c++ and java
• python singleton pattern: wrap in method generator.
• ‘copy’ module can introspect and make a copy of an object
•  lxml.builder import E :: ‘framework-y’

structural patterns

Adapter: wrapper to add a method.  Examples:

• sockets send/receive rather than read/write.
• socket._fileobject adopter
• crucial pattern for letting us re-use code

Bridge

• Gary Bernhart: “have modules that do nothing but persist information. Separate your business logic into functions that take data objects”

Composite: “if a module imports 3 major libs or frameworks, refactor into 2 modules each of which only imports 2 libraries”

Facade: an object that hides a structure.

Flyweight pattern: only create single instance of static data

Proxy

• An obj that wraps another obj, like adapter, but accepts the same methods as the wrapped class.
• ex: remote procedure calls

Decorator (not like the python decorators)

• tend to appear in ‘glue’ code

behavioral patterns

• Chain of responsibility
• Command: wrap, track history, enable undo
• Interpreter: domain specific languages
• Iterator/generator:  “most important addition of the 2000’s”
• Mediator
• Memento
• Observer
• State, template, visitor
• most of these turn up in bigger projects

Conclusion

Python programmers rarely meet the bigger patterns because python programmers tend to write smaller programs and scripts.  Frameworks have already implemented them.

Python for humans

presenter: @kennethreitz
works for Heroku

“open source encourages best practices”

zen of python:

• Explicit is better than implicit
• Simple is better than complex
• Complex is better than complicated
• If the implementation is hard to explain, it’s a bad idea

Barriers to entry

• HTTP request example. Problem in python: unclear which module to use.
• “HTTP should be as simple as a print statement
• “python needs more pragmatic packages”
• Requests library (http://docs.python-requests.org/en/latest/index.html)

Other python examples:

• Packaging & Dependencies
• Date/Times
• Unicode
• Testing
• Dependencies

Hitchhikers guide to python – an opinionated best practices guide to installing and using python.

Slides: https://speakerdeck.com/u/kennethreitz/p/python-for-humans

“There should be one – and preferably only one – obvious way to do something” – more python zen

Django-CMS

Presenter: Andrew Schowen – @andrewschoen

@django_CMS: https://www.django-cms.org/

Andrew’s demo: https://github.com/andrewschoen/django-cms-presentation

You can edit content (templates/plugins) within the context of your page

• create the pages in the menus at the same time.
• internationalized

Structure: templates, placeholders, menus

Plugins: Yes! Plugins are available.  You can also easily write custom plugins using other Django app/model

apphooks: direct through djangocms to the app. templates make for abstract skinning.  CMS app puts ‘polls’ in the menu.

Extensions and 3rd party apps

• django-filer: image manager
• cmsplugin-filer
• django-reversion: lets you version control page data
• cmsplugin-blog
• django-cms-redirects: lets you manage (303, 410) errors

Lightning Talks 2

• Bookie
  • Rick – @mitechie
  • @BookieBmarks
  • text searchable bookmark links
  • github.com/mitechie/breadability/
  • https://bmark.us/signup
• Data Science with Python
  • @sampathweb
  • Kaggle – making a sport out of data analysis
  • pandas – data frames/time series
  • sampathweb.com
• MaximinProject
  • people know poverty exists, but don’t know much about it or where it is.
  • human wealth index
  • non profit to build website to connect funders with groups on the ground
• Dependency Injection – A Python Patterns Encore
  • Brandon Rhodes
  • @brandon_rhodes
  • “if i define it, i can’t be wrong”
  • ex: urlopen – add a parameter to pass real object or some simple test function
  • pass in the callable function rather than chaining function calls.
• From Python Import Podcast
  • Mike
  • http://www.frompythonimportpodcast.com/
  • new podcasts are up, please resubscribe
  • website built with Responsive HTML
  • PyOhio episode
• Dynamic DNS
  • https://github.com/myano/linode-dyndns
• HTSQL
  • @catherinedevlin
  • htsql.org
  • URL:- SQL access to a healthcare database
  • API for python

I’m looking forward to catching the other amazing presentations I missed on pyvideo.org

Update: The videos have been posted to pyvideo.org

Choice Sets

I decided to try using choice sets for a few fields.  They are nice in that they let you configure a list of coded values.  However, getting at the long form descriptions was a puzzle to me initially.  Before I noticed the get_FOO_display() function, a friend pointed me to this Stack Overflow question.  The last entry was perfect for my needs:

def get_display(key, list):
    d = dict(list)
    if key in d:
        return d[key]
    return None

I suspect that with a little effort, I should only need to rely on get_FOO_display().  However, with a free standing choice list, get_display() may come in handy.

Dumping your data

During the early stages of fleshing out a model, one generates a nice set of data.  While one could manually recreate it, it quickly reaches the tipping point of tediousness.  In the course of developing your data model, there comes a time when you will want to modify your data structure in a way that will conflict with a previous version.  For me, this first came trying to add a BooleanField and getting the following error message: “no such column”.  I updated my model and ran syncdb, what do you mean no such column?!?  With some assistance, best I could figure is that Django didn’t know what to do with the previous rows.  Since I’m still in the very early stages of designing my app, I decided to dump and rebuild my database.

While it is not outside of the realm of possibility to manually re-enter the 4 rows (plus related entries) I had already accumulated, that can be a major pain to do so.  I know there are tools like South to help with data migrations, but that is overkill for my needs at this point.  Fortunately, Django provides a way to save off and reload your data.  This is what I worked out for my toy projects (having an sqlite database in database.db):

python manage.py dumpdata [appName] --indent 2 > dumpfile.json
rm database.db
python manage.py syncdb
python manage.py loaddata dumpfile.json

Delete the database, re-sync everything, load the file.  Magic.

python manage.py runserver

Took me a few missteps to remember this command to enable the development web service accept outside HTTP requests. Figured that was as good a sign as any to write this one down. Excerpt from the Django tutorial:

If you want to change the server’s IP, pass it along with the port. So to listen on all public IPs (useful if you want to show off your work on other computers), use:

python manage.py runserver 0.0.0.0:8000

Other learning mistakes…

• When setting up a view-template, I naively tried to pass MyModel.objects which does not return a queryset.  You need to call a method like .all() or .order_by(field).
• By having a URL in my main project with pattern “^myapp/$” – that trailing “$” prevented django from looking into my app’s urls.py file.

Other good resources

I stumbled upon this site a while ago: Django by Example which has several good examples including a To-Do list, a simple blog, a photo organizer, a forum, and a calendar.  I haven’t tried these yet on 1.4, but am planning to in the near future.

Another good jumping point I’d like to dive into further once I get more comfortable with Django: https://github.com/hjwp/Test-Driven-Django-Tutorial

Names, Objects, and Plummeting from the Cliff

This first video I found helpful in understanding how python handles basic variables (names) and objects through functions calls and reference counting.  This talk presents the concept in a very straight forward way that is very accessible, even to new programmers without any major background in computer science. Coming from another language where pass by reference is handled differently, I found this a valuable primer.  I highly recommend this video for anyone new to python.

http://pyvideo.org/video/542/pyohio-2011-names-objects-and-plummeting-from

Procedures, Objects, & Reusability

The second video is about procedural vs. object oriented python.  Brandon uses the gnarly implementation of httplib (and his quest to use the http parser) to illustrate the difficulty of understanding tightly coupled code, especially within an object.  He make an argument for times it is appropriate to use stand alone procedures vs. object methods that perform singular operations with clear inputs/outputs/expectations.  Having worked in a procedural environment with tight code coupling and limited to no object orientated paradigm, I have seen definite value to strictly procedural functions, either as an end point or transitional stage to OO.  I have found myself at times times simulating OO with stand alone functions and data structures.  There is a time and a place for good procedural programming.

http://pyvideo.org/video/521/pyohio-2011-procedures-objects-reusability-q

Squinting at Python Objects

The third video is a sort of follow up where Brandon explains a package he developed to help him safely examine the properties of objects that are immutable.  This one dives deeper into the technical realm, but is nevertheless interesting.  I have not presently worked with python code at this level, so I found this talk fascinating.

http://pyvideo.org/video/524/pyohio-2011-squinting-at-python-objects

Happy programming!

I cannot wait to brush up on my python, learn a few new things, and make the jump to 3.x.

• PyCon 2010:Persistent Graphs in Python with Neo4j (#140)
• PyCon 2010:Using Python to Create Robotic Simulations for Planetary Exploration (#159)
• PyCon 2010:What every developer should know about database scalability (#21)
• PyCon 2011: Python.MiroCommunity.org + UniversalSubtitles.org
• PyCon 2011: Greasing the Wheels of Exploration with Python
• PyCon 2011: Django: Pitfalls I Encountered and How to Avoid Them

I am constantly amazed at what people are doing with Python.