Not long ago, I showed you how to add tracing to a JavaScript application with Sentry. In this article, we’ll walk through using Sentry on the back end to track errors in a Python-Django application stack. 

Install and set up Python and Django

The first step is to create a Django application. We’re going to use Python 3, so start by installing Python 3 on your system. We’ll also need pip3. On Debian, I install them both by entering the command, sudo apt-get install python3 python3-pip. Once you have Python and pip installed you should get responses from the -V switch, as shown in Listing 1.

Listing 1. Make sure python3 and pip3 are available

$python3 -V
Python 3.11.0
$ pip3 -V
pip 22.3.1 from /usr/lib/python3/dist-packages/pip (python 3.11)

Next, install the Django framework and verify it is working, as shown in Listing 2.

Listing 2. Install Django

$ pip3 install Django
…
$ django-admin --version
4.1.3

Now we can start a new application using Django. Enter the command, django-admin startproject djangosentry. This creates a directory to hold our starter project. Now, CD into the project, /djangosentry. Django manages the application database, so we just need to fill it with the initial data needed for running a Django type. Enter, python3 manage.py migrate, where manage.py is the main Django script. After that completes, we’ll add a user by entering: python3 manage.py createsuperuser. We will need this user to log in. (Note that you will be prompted for a username and password—just use something you will remember.)

We need to tell Django to allow the current host. You can find out your IP address by pointing your browser to https://whatsmyip.com. Go to djangosentry/setting.py and add the IP address where the instance is located to the ALLOWED_HOSTS array; something like: ALLOWED_HOSTS = ['192.168.0.122']. Now, you can start the server by typing: python3 manage.py runserver 0.0.0.0:8000.

If you visit http://192.168.0.122:8000, you will see the Django welcome screen shown in Figure 1.

IDG

Note that there is also an admin console to add users and groups at  http://192.168.0.122:8000/admin.

Set up Sentry

Now that we have a running Django application, let’s set up Sentry to track errors. The first step is to create a Sentry account, which is free if you select the Developer option. After you have an account and log in, you’ll see the Sentry dashboard. The left-hand menu presents the main navigation for Sentry. Click on Projects and create a project with the Create Project button in the upper-right corner. 

IDG

Create and configure a Django project

A project gives us a central container for our tracing info. After you hit Create Project, you can select “Django” as the type and accept the default name of django-python.

After the project is created, you will be presented with the option to configure it. Copy the code snippet shown in Figure 3 and add it to your project.

IDG

Now that we have an application and a project, we can add the Sentry library. Return to the djangosentry project on your command line and type: pip install --upgrade sentry-sdk. This command adds the Sentry library to your project.

Now, we’ll import and configure the SDK into our djangosentry/settings.py file, which is the main configuration file for Django. This is just a few lines of code, but we need the data source name (DSN) for the project. The idea here is that the DSN will create the association between the Django project and the project in Sentry, so you can track project activity using the dashboard.

Now go to the djangosentry/djangosentry/settings.py file and add the code you copied to the bottom of the file. The result should look like what you see in Listing 3.

Listing 3. Add Sentry SDK and project DSN to settings.py

import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

sentry_sdk.init(
    dsn="<THE DSN YOU COPIED>",
    integrations=[
    DjangoIntegration(),

    ],
    traces_sample_rate=1.0,
    send_default_pii=True
)

Believe it or not, the Django project is ready to capture error data. In fact, the traces_sample_rate setting tells Sentry to also capture performance metrics—this value can be anywhere from 0 to 1.0. Note that setting your sample rate to 100% (that is, 1.0) can quickly exhaust your quota. Be sure to adjust your sample rate after collecting the initial performance data.

Error reporting with Sentry

Let’s create a quick error to see if Sentry is working. Go back into your settings.py file and remove the IP address from the allowed hosts. Then, restart the application and reload the webpage. You’ll get an error page like the one in Figure 4.

IDG

If you check the Django console output you’ll also find the error: [20/Nov/2022 11:17:40] "GET / HTTP/1.1" 400 69108. The Django server is returning an HTTP permission error 400.

Now, let’s look at the same error report from Sentry. By default, the email address you used to set up your Sentry account is subscribed to alerts. For the HTTP permission error, you will receive an alert similar to what you see in Figure 5.

IDG

Next, take a look at the Sentry dashboard. If you open the python-django project you’ll see the error reflected there, as shown in Figure 6.

IDG

You can click on the error in the dashboard to get details. You will see pretty much everything the system could know about what happened, from the user agent and timestamp to whether it was a handled exception (in this case, it was). 

Configuring the environment

Notice that the error in Figure 6 is tagged as environment: production. Lacking further configuration, Sentry assumed the error came from production. To manually set the environment, go to the djangosentry/settings.py file. In the sentry_sdk.init() function call, add an argument specifying an environment, as shown in Listing 4.

Listing 4. Adding an environment

sentry_sdk.init(
    dsn=”https://[email protected]/4504193108672512”,
    	integrations=[
        	DjangoIntegration(),
   	],
   	 traces_sample_rate=1.0,
    	send_default_pii=True,
    	environment="dev" # < ADD THIS
)

Now, when an error is reported, Sentry will assign it to the dev environment. You could set this string from an environment variable (for example called SENTRY_ENVIRONMENT). On your server, you set this variable as production, staging, or qa. On your local machine, set the variable to dev. You can then select from the dropdown list and filter by the environment in the dashboard.

Coding errors and issue tracking

Instead of crashing the example application, let’s first fix the ALLOWED_HOSTS entry by putting the correct IP address back in there. Then, let’s create a typical coding error by adding a Django route that spews an exception. Open the djangosentry/urls.py file and modify it to include the /error path, as shown in Listing 5.

Listing 5. Add an error path to urls.py

def trigger_error(request):
    raise Exception("Something has gone seriously wrong and you're coding skills are in question.")

urlpatterns = [
    path('admin/', admin.site.urls),
    path('error/', trigger_error)
]

Now, if you visit the localhost:8000/error route you’ll see the custom exception. You’ll notice that Sentry has by default grouped similar errors into issues. The new exception has triggered an alert email, but repeatedly causing the same error does not. You can see the same kind of organization on the Sentry dashboard, as shown in Figure 7.

By default, similar errors (or events) are grouped into issues. This is configurable and you can organize several events into a single issue.

IDG

Performance monitoring

Now that you have set up Sentry to track and report errors, you can set it up to also capture performance metrics to quickly identify latency and throughput issues.

IDG

Here, you can see the performance breakdown of the error path you added earlier. Sentry recently added a new feature to detect common performance problems to its application performance monitoring platform. N+1 database queries for Django projects are the first problem type that Sentry introduced for the new Performance Issues feature. As described in this tweet, the team that created Django used the new feature to detect an N+1 issue on Django’s documentation site. Django Fellow Carlton Gibson commented: “It was nice to see the N+1 queries pop-up in the Sentry issues list. A quick select_related and all was resolved, as it was easy to miss where these [N+1 queries] come up. I think it’s going to be quite handy.”

Conclusion

We’ve done a quick tour of some of Sentry’s capabilities in a Python-Django application context. There is quite a lot of power under the hood on Sentry’s dashboard, but perhaps the nicest thing of all is how easily you can use it to add error tracking to an application. All we had to do was obtain our DSN, make a small code change, and errors were being reported across the application with reasonable, configurable defaults.

With data more critical than ever to companies’ success, Python is spreading beyond the realm of data professionals and being adopted by business analysts and other less technical users. But what are the opportunities if you’re relatively new to Python and what best practices should you be aware of to ensure your success?

Data professionals are a precious commodity and in many organizations the demands of the business have outgrown the resources and capacity of data teams. At the same time, business analysts are running into the limits of what BI tools can do for them and looking for ways to do more advanced analytics. Python is the key to success here.

Python usage is growing fast. In a survey of more than 20,000 developers earlier this year, Python ranked second only to JavaScript in terms of popularity, and Python added 3.3 million net new users over the previous six months to reach 15.7 million users worldwide.

In recent years, the Python community has created new frameworks and packages that make the language more accessible to non-professional developers for advanced analytics, machine learning, and app development. Examples include NumPy, an open source Python library for numerical data; Prophet, for running forecasts, and H3, a project begun at Uber for manipulating geospatial data.

Python’s spread to non-professional developers isn’t without precedent. A similar pattern played out with the rise of self-service BI tools, and with business people learning to script their own Excel macros. The expanded use of Python will be even more impactful because the language itself is so capable.

Getting started with Python analytics

Business users often understand better than professional developers what specific insights will be most helpful to their business units, and there are several entry-level use cases where they can start putting Python to work. Here are three examples:

Correlation matrices

A correlation matrix is a table that shows the correlation coefficients for different variables. This can allow you to analyze different dimensions of a data set to determine if a person who exhibits behavior A, for example, is also likely to exhibit behavior B. Correlation matrices are useful for determining which items to place near to each other in a grocery store, or which additional items to offer when an ecommerce user is checking out.

Principal component analysis

Another possible starting point is principal component analysis, which can reduce the size of a noisy data set and determine which attributes have the most predictive power for a given outcome. If a company sells mortgages, for example, a principal component analysis can reveal which demographic factors (income, ZIP code, marital status, etc.) are most predictive of a sale, helping to target campaigns and offers.

Forecasting

Another common problem for businesses is forecasting. Think of predicting customer demand, sales, or revenue, which all mature businesses need to do. Building forecasts is a way to explore predictive analytics, using open source libraries such as Prophet or Scikit-Learn in Python. 

Great power, as they say, brings great responsibility, and there are best practices that new Python users should employ to ensure that the applications they build are robust and secure.

Python care and feeding

One issue is maintaining Python packages to ensure that dependencies are properly managed. Anaconda is helpful here, because it greatly simplifies package management and deployment. With Snowflake’s Snowpark for Python, we pre-install the most popular Python packages from the Anaconda defaults channel into our Python runtime so they don’t have to be installed manually. We’ve also integrated the Conda package manager into Snowpark to manage Python packages and their dependencies.

Like any data project, there are security and governance issues to be aware of, but modern cloud data platforms provide a runtime that is already set up and configured, and users can take advantage of the security and governance capabilities built into those platforms. For example, the Python runtime in Snowpark disallows external network access by default to protect against common security concerns such as data exfiltration. Using a pre-configured secure Python runtime like Snowpark is much easier for novice Python users compared to creating and maintaining your own environments or containers.

It’s early days still, and over time I expect additional Python tools and resources aimed specifically at non-professional developers to emerge. One area that needs to evolve is the methods by which Python users can share the outputs of their work with colleagues who don’t want to learn the language themselves. Snowflake’s purchase of Streamlit was intended in part to address this. The open source tool allows data teams to build applications that bring data to life visually for non-technical users. Python itself is a powerful language for building applications, so its use in building data applications for end users will make the language even more widely adopted. 

To get started, RealPython offers a comprehensive beginner’s guide to Python, and Full Stack Python links to many resources here. The Python Software Foundation has an active community where experienced users provide advice and answer questions for all ability levels. 

If you’re a Snowflake user, read about our Snowpark developer environment here, which natively supports Python development. You can also join one of the many Snowflake community user groups worldwide, which arrange meetups to discuss technical developments and opportunities.

Torsten Grabs is director of product management at Snowflake.

—

New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to [email protected].

The Python programming language releases new versions yearly, with a feature-locked beta release in the first half of the year and the final release toward the end of the year.

Python 3.11 has just been released, and developers are encouraged to try out this latest version on non-production code, both to verify that it works with your programs and to get an idea of whether your code will benefit from its performance enhancements.

Here’s a rundown of the most significant new features in Python 3.11 and what they mean for Python developers.

Speed improvements

Many individual performance improvements landed in Python 3.11, but the single biggest addition is the specializing adaptive interpreter. Since an object’s type rarely changes, the interpreter now attempts to analyze running code and replace general bytecodes with type-specific ones. For instance, binary operations (add, subtract, etc.) can be replaced with specialized versions for integers, floats, and strings.

Python function calls also require less overhead in Python 3.11. Stack frames for function calls now use less memory and are more efficiently designed. Also, while recursive calls aren’t tail-optimized (which probably isn’t possible in Python, anyway), they are more efficient than in previous versions. The Python interpreter itself also starts faster, and core modules needed for the Python runtime are stored and loaded more efficiently.

According to the official Python benchmark suite, Python 3.11 runs around 1.25 times faster than Python 3.10. Note that this speedup is an aggregate measure. Some things run much faster, but many others run only slightly faster or about the same. Still, the best part about these improvements is that they come for free. You don’t need to make any code changes for Python programs to take advantage of Python 3.11’s speedups.

Enhanced error information

Another immediately useful feature in Python 3.11 is more detailed error messages. Python 3.10 already had better error reporting, thanks to the new parser used in the interpreter. Now, Python 3.11 expands on that by providing detailed feedback about what specific part of a given expression caused an error.

Consider the following code, which throws an error:

x = [1,2,3]
z = x[1][0]

In Python 3.10, we’d receive the following error message, which is not very helpful:

  File "C:\Python311\code.py", line 2, in <module>
    z = x[1][0]
TypeError: 'int' object is not subscriptable

Rather than leave us wondering which int is not scriptable, the error trace in Python 3.11 points to the exact part of the line that generates the error:

  File "C:\Python311\code.py", line 2, in <module>
    z = x[1][0]
        ~~~~^^^
TypeError: 'int' object is not subscriptable

Now, there is no ambiguity about where the problem lies.

Exception improvements

Exceptions, Python’s error-handling mechanism, have received many new features in Python 3.11:

• Multiple exceptions can be raised and handled at once with the new except* syntax and the new ExceptionGroup exception type. This allows the elegant handling of issues where multiple errors can be raised together, such as when dealing with asynchronous or concurrent methods or when dealing with multiple failures when retrying an operation.
• “Zero-cost” exceptions: Exceptions now have no cost to a program unless they are actually raised. This means the default path for a try/except block is faster and uses less memory.
• The time needed to catch an exception has been reduced by around 10%.
• Exceptions can be enriched with contextual notes, separate from the text of the exception itself.

Typing improvements

Python’s type-hinting features make larger codebases easier to manage and analyze, and have increased significantly with each revision since Python 3.5. Python 3.11 brings in several new type-hinting additions.

The Self type

Class methods that return self previously required obtuse and verbose annotations to be useful. typing.Self lets you annotate the return value of a class method as, simply, Self. You get useful and predictable results from your analysis tools for such methods.

Arbitrary string literal type

Previously, type annotations had no way to indicate a given variable needed to be a string literal—that is, a string defined in source code. The new typing.LiteralString annotation fixes that. Using the new annotation, linters can test for a variable being either a string defined in source or a new string composed of only source-defined strings.

Dataclass transforms

Since Python 3.7, dataclasses have made it easier to define classes that followed common patterns for creating properties based on their initialization parameters. But there was no standard mechanism for allowing things that behaved like dataclasses (but weren’t dataclasses themselves) to use type annotations to declare their behavior. Dataclass transforms adds the typing.dataclass_transform decorator to indicate how a given function, class, or metaclass behaves like a dataclass.

Variadic generics

The original proposal for type hints included TypeVar, a way to specify a generic function using a single parameterized type—for example, a type T that could be an int or a float. Python 3.11 adds TypeVarTuple, or “variadic generics,” which you can use to specify a placeholder for not just one type but a series of them, expressed as a tuple. This would be especially useful in libraries like NumPy, where you could perform ahead-of-time checks for errors like whether a supplied array was the correct shape.

TOML read-only support in stdlib

Python uses TOML, or Tom’s Obvious Minimal Language, as a configuration format (as in pyproject.toml), but doesn’t expose the ability to read TOML-format files as a standard library module. Python 3.11 adds tomllib to address that problem. Note that tomllib doesn’t create or write TOML files; for that you need a third-party module like Tomli-W or TOML Kit.

Atomic grouping and speedups for regex

Python’s re module, for working with regular expressions, has lacked a few features found in other implementations of regular expressions. One is atomic grouping, widely supported in other languages. Python 3.11 now supports this pattern using the common syntax for atomic groupings (as an example, (?>...)).

The re module’s pattern matching engine has also been rewritten somewhat, and runs about 10% faster.

Removing ‘dead batteries’ from the standard library

PEP 594 kicked off an effort to remove many so-called dead batteries, or modules that are obsolete or unmaintained, from the Python standard library. As of Python 3.11, those libraries are marked as deprecated but not yet removed; they will be removed entirely in Python 3.13.

Other Python 3.11 additions, fixes, and changes

Many more smaller improvements also landed in Python 3.11:

• Python objects require less memory, as their namespaces are now lazily created, and their namespace dictionaries now share keys whenever possible.
• Dictionaries where all keys are Unicode no longer need to store hashes, thus reducing the size of the dictionary and allowing more cache efficiency.
• The CPython runtime, the reference interpreter for Python, now has experimental support for being compiled to WebAssembly. This may aid the future development of projects like PyScript, which allow a Wasm-compiled Python runtime to operate in the browser.