What is Python? Enterprise, Data Science, and the Backend of the Web
Python runs Instagram, Dropbox, Netflix, Spotify, and Reddit. It trains the models that power modern AI. It processes petabytes of data in enterprise pipelines. And I ignored it for years because I thought it was just for academics. Here is what changed my mind.
Every developer I respected told me to learn Python. Not once. Over and over. Beginners heard it in tutorials. My senior engineer friends said it over drinks. Online communities treated it like gospel. "Python is perfect for beginners." "Python is so readable." "You really should learn Python."
I was a PHP developer. I knew JavaScript. I had MySQL. I wrote the occasional Python script when something needed automating, but I never took it seriously. It looked like a toy. Something for academics and data people, not for building real production systems.
Then in 2017, I started offering free hosting to open-source projects I liked. One of them was a Discord bot, an IdleRPG game written in Python on top of PostgreSQL. The main developer needed performance optimization, async migration, and new features. I wanted to help. So I actually learned Python this time.
That bot grew from 3 Discord servers to 50,000 in about a year, then to 100,000+ on a single dedicated server. I watched Python handle massive concurrent workloads on hardware that should not have been able to manage it. I stopped being skeptical and started being a convert.
But the Discord bot is only part of the story. Python is not a niche language for bots and scripts. It is the backbone of some of the largest infrastructure on the internet, the dominant language in data science and machine learning, and the tool that powers analytics pipelines processing petabytes of data daily. Understanding what Python is means understanding why it ended up doing all of that.
What Python Actually Is
Python is a high-level, interpreted programming language created by Guido van Rossum. The first version shipped in 1991. Van Rossum named it after Monty Python's Flying Circus, which tells you something about the design philosophy: the language was supposed to be fun to use.
The core design decision is readability. Python uses indentation to define code blocks instead of curly braces. This sounds minor until you have been staring at nested PHP callbacks for three hours. In Python, the visual structure of the code mirrors the logical structure. You cannot write visually confusing code without the interpreter refusing to run it.
Python is interpreted, meaning there is no separate compilation step. You write code in .py files and run them. The CPython interpreter reads your source code, compiles it to bytecode internally, and executes that bytecode on a virtual machine. This is why Python starts fast but runs slower than compiled languages for CPU-heavy work. That tradeoff is intentional and the reasons for it matter, which I will get to.
Python is dynamically typed. Variables do not require type declarations. The interpreter figures out types at runtime. This speeds up development significantly because you write less boilerplate, but it means type errors surface at runtime rather than compile time. Python 3.5 introduced optional type hints to address this, and the ecosystem around them has grown into something genuinely useful for large codebases.
Everything in Python is an object. Integers, strings, functions, classes. All of them are objects with methods and attributes. This gives Python a consistency that other languages lack. When you call "hello".upper(), you are calling a method on a string object. When you call (42).bit_length(), you are calling a method on an integer object. The mental model is uniform throughout the language.
The standard library is what Python developers call "batteries included." Networking, file I/O, JSON parsing, date handling, cryptography, database connections, HTML parsing, regular expressions: all of it ships with the language. You can build production-grade tools without installing a single third-party package.
The Syntax
Python's syntax is worth spending time on because it is the reason the language scales from beginner tutorial to NASA mission planning code.
name = "Traven"
age = 28
languages = ["Python", "JavaScript", "PHP"]
print(f"My name is {name}, I am {age} years old.")
print(f"I know {len(languages)} languages.")
for lang in languages:
print(lang)
F-strings (formatted string literals, introduced in Python 3.6) let you embed expressions directly in strings with {} syntax. len(languages) calls the built-in function that returns the length of any iterable. The for loop iterates over the list without requiring an index variable.
Classes in Python look like this:
from dataclasses import dataclass
from typing import Optional
@dataclass
class User:
id: int
username: str
email: str
level: int = 1
gold: int = 0
is_active: bool = True
bio: Optional[str] = None
def display_name(self) -> str:
return f"{self.username} (Level {self.level})"
def award_gold(self, amount: int) -> None:
if amount <= 0:
raise ValueError(f"Gold amount must be positive, got {amount}")
self.gold += amount
user = User(id=1, username="Traven", email="[email protected]", level=5, gold=100)
print(user.display_name())
user.award_gold(50)
print(user.gold)
The @dataclass decorator auto-generates __init__, __repr__, and __eq__ from the class fields. You define fields with type annotations and optional default values. This is modern Python. It removes the boilerplate that used to make class definitions tedious while keeping the explicitness that makes code readable.
The Optional[str] type hint says this field can be a string or None. The -> str annotation on display_name says the method returns a string. These are not enforced at runtime unless you use a type checker like mypy, but they make code significantly easier to understand and catch bugs before they ship.
The Ecosystem
The Python Package Index (PyPI) hosts over 500,000 packages. Every domain of software development has multiple mature Python libraries. This did not happen by accident. The scientific community adopted Python in the late 1990s and early 2000s, which seeded the data ecosystem. Web developers followed. Machine learning researchers followed. By the time deep learning exploded around 2012, Python was already the dominant scientific computing language, and it absorbed the entire ML wave.
Package management in Python uses pip:
pip install requests pandas numpy fastapi
For project isolation, you use virtual environments. Each project gets its own isolated Python environment with its own packages so dependency versions do not conflict between projects:
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
For managing Python versions across projects, the standard tool is pyenv:
pyenv install 3.12.0
pyenv local 3.12.0
python --version
In production, you almost always use a requirements.txt file or pyproject.toml to pin exact dependency versions. Unpinned dependencies are how production deployments break in ways that are impossible to reproduce locally.
fastapi==0.115.0
uvicorn==0.32.0
sqlalchemy==2.0.36
asyncpg==0.30.0
pydantic==2.9.2
python-jose==3.3.0
passlib==1.7.4
Modern Python projects increasingly use pyproject.toml with tools like poetry or uv for dependency management, but requirements.txt is still common in production environments.
The Companies Running Python at Scale
This is the part where the "Python is slow" crowd goes quiet.
Instagram runs the largest deployment of Django on the planet. Their backend serves over 2 billion active users. When Facebook acquired Instagram in 2012, the entire backend was Python. It still is. Their engineering team has published detailed breakdowns of how they run Python at that scale, including custom CPython optimizations and aggressive caching strategies.
Dropbox built their core product almost entirely in Python. Guido van Rossum, Python's creator, worked at Dropbox from 2013 to 2019 to help scale their Python infrastructure to hundreds of millions of users syncing petabytes of files. The desktop client was Python. The backend was Python. When they eventually migrated parts of the client to Rust for performance, they did so surgically, keeping Python where it made sense.
Spotify uses Python for data pipelines, analytics, and backend services. Their music recommendation system, which processes billions of plays to surface what you might want to hear next, runs substantial Python workloads. Their data engineering team has open-sourced several Python tools including Luigi, a pipeline framework that became an industry standard.
Netflix runs Python throughout their infrastructure. Their chaos engineering tool Chaos Monkey has Python components. They use Python for CDN optimization, deployment automation, and the data science work that powers content recommendations. Netflix engineers have given conference talks specifically about Python's role in their ML infrastructure.
Reddit rewrote their entire platform from Lisp to Python in 2005. They open-sourced the Reddit codebase, which became one of the most studied large Python applications in existence. The platform still runs Python today, handling billions of page views monthly. Their transition is documented and worth reading if you want to understand what Python looks like at scale.
YouTube built significant portions of their application layer in Python from early in the company's history. The video serving infrastructure runs C++, but the Python application layer handles enormous traffic. Google acquired YouTube in 2006 and continued the Python investment.
Pinterest built their backend on Python with Django. They serve hundreds of millions of users and have published engineering blog posts about scaling Python to handle their image processing and recommendation workloads.
NASA uses Python for scientific computing, mission planning, and data analysis. They have contributed Python code to public repositories. The software that processed data from the Mars Ingenuity helicopter included Python components.
The pattern across all of these is the same. Python is not fast at CPU-bound computation in a single thread. Python is excellent at I/O-bound work, at orchestrating distributed systems, at data pipelines where the heavy computation happens in optimized C libraries underneath Python's surface. The Instagram engineers are not running naively interpreted Python. They are using Python as a high-level language that calls into optimized native code for the bottlenecks.
Python and Data Science
Python owns data science. This is not an opinion. Look at any job posting for a data scientist, machine learning engineer, or data engineer. Python is listed first on essentially all of them.
The reason is the scientific computing stack that was built in Python over two decades: NumPy, Pandas, Matplotlib, Scikit-learn, TensorFlow, PyTorch, and Jupyter. These libraries represent thousands of person-years of work, and they all interoperate through a shared data model based on NumPy arrays.
NumPy
NumPy provides N-dimensional arrays and mathematical operations on them. The array operations are implemented in C with BLAS (Basic Linear Algebra Subprograms) under the hood, so they run at speeds close to hand-optimized C code while you write Python.
import numpy as np
data = np.array([14, 23, 8, 42, 17, 31, 9, 55, 26, 38])
print(f"Mean: {np.mean(data):.2f}")
print(f"Std Dev: {np.std(data):.2f}")
print(f"Min: {np.min(data)}, Max: {np.max(data)}")
above_average = data[data > np.mean(data)]
print(f"Values above average: {above_average}")
normalized = (data - np.mean(data)) / np.std(data)
print(f"Normalized: {normalized.round(2)}")
data[data > np.mean(data)] is boolean indexing. NumPy evaluates data > np.mean(data) as an array of booleans, then uses that as a mask to select elements. This runs in compiled C code, not Python loops. On a 10-element array the difference is invisible. On a 10-million-element array it is the difference between milliseconds and minutes.
Pandas
Pandas builds on NumPy to provide the DataFrame, a table-like data structure with labeled rows and columns. It is the tool data scientists use for cleaning, transforming, and analyzing data from files, databases, or APIs.
import pandas as pd
import numpy as np
np.random.seed(42)
n_records = 10000
df = pd.DataFrame({
'user_id': range(1, n_records + 1),
'age': np.random.randint(18, 65, n_records),
'country': np.random.choice(['US', 'UK', 'DE', 'FR', 'JP', 'CA'], n_records),
'plan': np.random.choice(['free', 'basic', 'pro', 'enterprise'], n_records, p=[0.4, 0.3, 0.2, 0.1]),
'monthly_spend': np.random.exponential(scale=50, size=n_records).round(2),
'churn': np.random.choice([0, 1], n_records, p=[0.85, 0.15])
})
df.loc[df['plan'] == 'free', 'monthly_spend'] = 0
print(df.head())
print(f"\nShape: {df.shape}")
print(f"\nChurn rate: {df['churn'].mean():.1%}")
plan_analysis = df.groupby('plan').agg(
user_count=('user_id', 'count'),
avg_spend=('monthly_spend', 'mean'),
churn_rate=('churn', 'mean'),
total_revenue=('monthly_spend', 'sum')
).round(2)
print("\nPlan Analysis:")
print(plan_analysis)
high_value = df[
(df['monthly_spend'] > df['monthly_spend'].quantile(0.75)) &
(df['churn'] == 0) &
(df['plan'].isin(['pro', 'enterprise']))
].copy()
print(f"\nHigh-value retained users: {len(high_value)}")
print(f"Average spend: ${high_value['monthly_spend'].mean():.2f}")
print(f"Country distribution:\n{high_value['country'].value_counts()}")
df.groupby('plan').agg(...) groups rows by the plan column, then computes the specified aggregations for each group. The result is a new DataFrame with plan as the index and your computed columns. This replaces what used to require complex SQL queries and gives you results you can immediately chain into further analysis.
Machine Learning with Scikit-learn
Scikit-learn provides a consistent API for classical machine learning algorithms. Every estimator (classifier, regressor, transformer) uses the same fit() / predict() / transform() interface. You can swap algorithms by changing a single line.
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.pipeline import Pipeline
np.random.seed(42)
n = 5000
df = pd.DataFrame({
'age': np.random.randint(18, 70, n),
'tenure_months': np.random.randint(1, 120, n),
'monthly_charges': np.random.uniform(20, 150, n),
'support_tickets': np.random.poisson(2, n),
'contract_type': np.random.choice(['monthly', 'annual', 'biannual'], n),
'payment_method': np.random.choice(['credit', 'debit', 'bank', 'check'], n),
})
churn_prob = (
0.3 +
(df['support_tickets'] > 3).astype(float) * 0.2 +
(df['contract_type'] == 'monthly').astype(float) * 0.15 +
(df['tenure_months'] < 12).astype(float) * 0.1 -
(df['tenure_months'] > 60).astype(float) * 0.15
).clip(0, 1)
df['churned'] = (np.random.random(n) < churn_prob).astype(int)
le = LabelEncoder()
df['contract_encoded'] = le.fit_transform(df['contract_type'])
df['payment_encoded'] = le.fit_transform(df['payment_method'])
feature_cols = ['age', 'tenure_months', 'monthly_charges', 'support_tickets',
'contract_encoded', 'payment_encoded']
X = df[feature_cols]
y = df['churned']
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
models = {
'Logistic Regression': Pipeline([
('scaler', StandardScaler()),
('clf', LogisticRegression(random_state=42, max_iter=1000))
]),
'Random Forest': RandomForestClassifier(
n_estimators=100, max_depth=10, random_state=42, n_jobs=-1
),
'Gradient Boosting': GradientBoostingClassifier(
n_estimators=100, learning_rate=0.1, max_depth=4, random_state=42
)
}
for name, model in models.items():
cv_scores = cross_val_score(model, X_train, y_train, cv=5, scoring='roc_auc')
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]
print(f"\n{'='*40}")
print(f"Model: {name}")
print(f"CV AUC: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})")
print(f"Test AUC: {roc_auc_score(y_test, y_prob):.4f}")
print(classification_report(y_test, y_pred, target_names=['Retained', 'Churned']))
The Pipeline wraps a scaler and a classifier together so they fit and transform as a unit. This prevents data leakage, a common mistake where you accidentally let test data influence your training preprocessing. Cross-validation with cross_val_score gives you a more reliable performance estimate than a single train/test split.
The n_jobs=-1 parameter tells RandomForestClassifier to use all available CPU cores for training. This is how you push Python beyond single-threaded limitations for CPU-bound work: you use a library that does the parallel work in C.
Deep Learning with PyTorch
PyTorch is the framework most researchers and engineers reach for in 2024. It uses dynamic computation graphs, which makes debugging model architectures far less painful than TensorFlow's early static graph approach.
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
import numpy as np
torch.manual_seed(42)
np.random.seed(42)
n_samples = 10000
n_features = 20
X = np.random.randn(n_samples, n_features).astype(np.float32)
true_weights = np.random.randn(n_features).astype(np.float32)
y = (X @ true_weights + np.random.randn(n_samples).astype(np.float32) * 0.5 > 0).astype(np.float32)
split = int(0.8 * n_samples)
X_train_t = torch.FloatTensor(X[:split])
y_train_t = torch.FloatTensor(y[:split])
X_test_t = torch.FloatTensor(X[split:])
y_test_t = torch.FloatTensor(y[split:])
train_dataset = TensorDataset(X_train_t, y_train_t)
train_loader = DataLoader(train_dataset, batch_size=256, shuffle=True)
class BinaryClassifier(nn.Module):
def __init__(self, input_dim: int, hidden_dims: list[int], dropout: float = 0.3):
super().__init__()
layers = []
prev_dim = input_dim
for hidden_dim in hidden_dims:
layers.extend([
nn.Linear(prev_dim, hidden_dim),
nn.BatchNorm1d(hidden_dim),
nn.ReLU(),
nn.Dropout(dropout)
])
prev_dim = hidden_dim
layers.append(nn.Linear(prev_dim, 1))
self.network = nn.Sequential(*layers)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.network(x).squeeze(1)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = BinaryClassifier(input_dim=n_features, hidden_dims=[128, 64, 32]).to(device)
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=50)
criterion = nn.BCEWithLogitsLoss()
for epoch in range(50):
model.train()
total_loss = 0
correct = 0
total = 0
for X_batch, y_batch in train_loader:
X_batch, y_batch = X_batch.to(device), y_batch.to(device)
optimizer.zero_grad()
logits = model(X_batch)
loss = criterion(logits, y_batch)
loss.backward()
optimizer.step()
total_loss += loss.item()
preds = (torch.sigmoid(logits) > 0.5).float()
correct += (preds == y_batch).sum().item()
total += y_batch.size(0)
scheduler.step()
if (epoch + 1) % 10 == 0:
model.eval()
with torch.no_grad():
test_logits = model(X_test_t.to(device))
test_preds = (torch.sigmoid(test_logits) > 0.5).float()
test_acc = (test_preds == y_test_t.to(device)).float().mean().item()
train_acc = correct / total
print(f"Epoch {epoch+1:3d} | Loss: {total_loss/len(train_loader):.4f} | "
f"Train Acc: {train_acc:.4f} | Test Acc: {test_acc:.4f}")
The BinaryClassifier class extends nn.Module, which is PyTorch's base class for all neural network components. forward() defines the computation that happens when you pass input through the model. nn.Sequential chains layers in order. BCEWithLogitsLoss combines a sigmoid activation with binary cross-entropy, which is numerically more stable than applying sigmoid and then cross-entropy separately.
loss.backward() computes gradients through automatic differentiation. PyTorch tracks all tensor operations in a computation graph and walks backward through it to compute how each parameter contributed to the loss. optimizer.step() updates parameters using those gradients.
This is what the AI/ML industry runs on. When you use a product that has any ML feature, there is a high probability the model behind it was trained or prototyped in Python with PyTorch or TensorFlow.
Python as a Web Backend
I will cover Django and Flask in dedicated articles because they each deserve the space. But understanding Python's role as a web backend language requires knowing what the options are.
Django is the full-stack framework. It ships with an ORM, an admin panel, authentication, form validation, and a templating engine. Instagram runs Django. Pinterest runs Django. When you need to move fast on a complex application and want batteries included, you reach for Django.
Flask is the micro-framework. It gives you routing and request handling and almost nothing else. You compose the rest yourself from extensions. When you need a lightweight API service or want full control over your stack, Flask is the choice.
FastAPI is the modern high-performance option. It uses Python's type hints to auto-generate request validation, response serialization, and API documentation. It runs on ASGI instead of WSGI, which means true async request handling and significantly better performance under concurrent load. If you are building a new Python API in 2024, FastAPI is the most compelling choice.
Here is a FastAPI production example to demonstrate what modern Python web development looks like:
from contextlib import asynccontextmanager
from datetime import datetime, timedelta
from typing import Annotated
import asyncpg
from fastapi import Depends, FastAPI, HTTPException, status
from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
from jose import JWTError, jwt
from passlib.context import CryptContext
from pydantic import BaseModel, EmailStr
SECRET_KEY = "your-secret-key-stored-in-env"
ALGORITHM = "HS256"
ACCESS_TOKEN_EXPIRE_MINUTES = 30
pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")
class UserCreate(BaseModel):
username: str
email: EmailStr
password: str
class UserResponse(BaseModel):
id: int
username: str
email: str
created_at: datetime
class Token(BaseModel):
access_token: str
token_type: str
db_pool: asyncpg.Pool | None = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global db_pool
db_pool = await asyncpg.create_pool(
"postgresql://user:pass@localhost/myapp",
min_size=5,
max_size=20
)
yield
await db_pool.close()
app = FastAPI(title="My API", version="1.0.0", lifespan=lifespan)
async def get_db():
async with db_pool.acquire() as conn:
yield conn
def verify_password(plain_password: str, hashed_password: str) -> bool:
return pwd_context.verify(plain_password, hashed_password)
def create_access_token(data: dict, expires_delta: timedelta | None = None) -> str:
to_encode = data.copy()
expire = datetime.utcnow() + (expires_delta or timedelta(minutes=15))
to_encode.update({"exp": expire})
return jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)
async def get_current_user(
token: Annotated[str, Depends(oauth2_scheme)],
conn: Annotated[asyncpg.Connection, Depends(get_db)]
):
credentials_exception = HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Could not validate credentials",
headers={"WWW-Authenticate": "Bearer"},
)
try:
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
user_id: int = payload.get("sub")
if user_id is None:
raise credentials_exception
except JWTError:
raise credentials_exception
user = await conn.fetchrow("SELECT * FROM users WHERE id = $1", user_id)
if user is None:
raise credentials_exception
return user
@app.post("/users", response_model=UserResponse, status_code=201)
async def create_user(
user_data: UserCreate,
conn: Annotated[asyncpg.Connection, Depends(get_db)]
):
existing = await conn.fetchrow(
"SELECT id FROM users WHERE email = $1 OR username = $2",
user_data.email, user_data.username
)
if existing:
raise HTTPException(status_code=400, detail="Username or email already taken")
hashed_password = pwd_context.hash(user_data.password)
user = await conn.fetchrow(
"""
INSERT INTO users (username, email, password_hash, created_at)
VALUES ($1, $2, $3, $4)
RETURNING id, username, email, created_at
""",
user_data.username, user_data.email, hashed_password, datetime.utcnow()
)
return dict(user)
@app.post("/token", response_model=Token)
async def login(
form_data: Annotated[OAuth2PasswordRequestForm, Depends()],
conn: Annotated[asyncpg.Connection, Depends(get_db)]
):
user = await conn.fetchrow(
"SELECT * FROM users WHERE username = $1", form_data.username
)
if not user or not verify_password(form_data.password, user["password_hash"]):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Incorrect username or password",
)
access_token = create_access_token(
data={"sub": user["id"]},
expires_delta=timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)
)
return {"access_token": access_token, "token_type": "bearer"}
@app.get("/users/me", response_model=UserResponse)
async def read_current_user(current_user: Annotated[dict, Depends(get_current_user)]):
return current_user
This is a production-grade FastAPI setup with JWT authentication, asyncpg connection pooling, dependency injection, and Pydantic models for request validation. FastAPI auto-generates OpenAPI documentation from the type hints and Pydantic models. Visit /docs on any running FastAPI app and you get an interactive API explorer for free.
The Discord Bot: Async Python in Production
This is the experience that converted me. Before that Discord bot, I knew Python syntax. After it, I understood what Python could do under load.
I covered the full story in the previous version of this article, including the progression from synchronous blocking code to async connection pooling to caching to asyncio.gather() for concurrent queries. If you want to read the deep dive on how we went from 3 servers to 100,000+ and what the code looked like at each stage, that article is here.
The short version: async Python, when written correctly, handles concurrent I/O workloads extremely well. The Discord bot processed millions of commands per day on a single dedicated server. The bottleneck was never Python's runtime. It was database query design and network I/O, problems that async programming solves.
Python's Real Weaknesses
I am not going to pretend Python is perfect. It has genuine limitations that matter.
The GIL. The Global Interpreter Lock is a mutex in CPython that prevents multiple threads from executing Python bytecode simultaneously. One Python thread runs at a time. For CPU-bound work in a single process, Python's threading model is essentially useless. You cannot saturate multiple CPU cores with Python threads running pure Python code.
The workarounds are real and production-proven. The multiprocessing module spawns separate processes, each with their own GIL, and they run in parallel. Libraries like NumPy, PyTorch, and scikit-learn drop the GIL when calling into C extensions, so they use multiple cores even within a single process. For I/O-bound work, asyncio sidesteps the GIL entirely because the bottleneck is waiting for network/disk, not CPU.
Python 3.13 introduced experimental support for a free-threaded mode without the GIL. It is not production-ready yet, but it signals that the core team takes the limitation seriously.
Speed. CPython runs Python bytecode slowly compared to compiled languages. A CPU-intensive Python loop runs 10 to 100 times slower than equivalent C or Rust code. This is the price of dynamic typing and the interpreter overhead.
The practical impact depends on what you build. Instagram's Django backend is I/O-bound. Its performance is dominated by database queries and network latency. Python's runtime speed is irrelevant because the bottleneck is not CPU. In contrast, a video encoding service or a physics simulation would be a terrible fit for pure Python.
Mobile. Python has no viable path to native mobile applications. Kivy exists and developers use it, but the resulting apps look and feel out of place and the ecosystem is sparse. If you need to ship iOS or Android apps, Python is not the tool.
Frontend. Python does not run in the browser natively. WebAssembly opens some possibilities through projects like Pyodide, but this is not mainstream Python development. For anything that runs in the browser, you are writing JavaScript or TypeScript.
Startup time. Python applications start slowly compared to compiled executables. This matters for CLI tools and serverless functions where cold start time is part of the user experience. Go and Rust programs start in microseconds. Python programs start in tens to hundreds of milliseconds. For a long-running web server this is irrelevant. For a CLI tool you run dozens of times per day, it is annoying.
Modern Python Features Worth Knowing
Python has evolved significantly since version 2. If you learned Python more than a few years ago and have not kept up, here are the features that changed how production code is written.
Type hints (Python 3.5+, dramatically improved in 3.9 and 3.10):
from typing import TypeVar, Generic
from collections.abc import Callable, Awaitable
T = TypeVar('T')
def process_items(
items: list[int],
transform: Callable[[int], int],
filter_fn: Callable[[int], bool] | None = None
) -> list[int]:
result = [transform(item) for item in items]
if filter_fn:
result = [item for item in result if filter_fn(item)]
return result
result = process_items(
items=[1, 2, 3, 4, 5],
transform=lambda x: x ** 2,
filter_fn=lambda x: x > 5
)
print(result)
Structural pattern matching (Python 3.10+):
from dataclasses import dataclass
@dataclass
class Point:
x: float
y: float
def classify_point(point: Point) -> str:
match point:
case Point(x=0, y=0):
return "origin"
case Point(x=0, y=y):
return f"on y-axis at {y}"
case Point(x=x, y=0):
return f"on x-axis at {x}"
case Point(x=x, y=y) if x == y:
return f"on diagonal at {x}"
case Point(x=x, y=y):
return f"at ({x}, {y})"
def handle_api_response(response: dict) -> str:
match response:
case {"status": "success", "data": data}:
return f"Got data: {data}"
case {"status": "error", "code": 404, "message": msg}:
return f"Not found: {msg}"
case {"status": "error", "code": code, "message": msg}:
return f"Error {code}: {msg}"
case {"status": "error"}:
return "Unknown error"
case _:
return "Unexpected response format"
Walrus operator (Python 3.8+):
import re
logs = [
"ERROR: connection timeout after 30s",
"INFO: user logged in",
"ERROR: database query failed",
"WARNING: high memory usage",
"ERROR: rate limit exceeded"
]
errors = [
match.group(1)
for line in logs
if (match := re.search(r"ERROR: (.+)", line))
]
print(errors)
The walrus operator := assigns and returns a value in the same expression. This removes the need for a separate variable assignment before the condition check in list comprehensions and while loops.
The Python 2 to Python 3 Saga
Python 3 launched in December 2008. The new version broke backward compatibility intentionally. print became a function. unicode became the default string type. Integer division returned floats. range() returned a lazy iterator instead of a list.
These were the right decisions. Python 2's handling of text and bytes was a constant source of bugs in any application that touched non-ASCII characters. Python 3 fixed the underlying design problem.
The migration took over a decade and became one of the messiest transitions in language history. Python 2 was officially supported until January 1, 2020, meaning the community ran two incompatible versions of the same language in parallel for twelve years. Entire ecosystems split between Python 2 and Python 3 support. Libraries refused to migrate until their dependencies migrated first. Companies delayed migration because their dependencies had not migrated yet. The deadlock held for years.
I watched this play out in real time. The Discord bot I worked on had Python 3 as a hard requirement from the start because asyncio only existed in Python 3. We never touched Python 2. But plenty of the enterprise Python code I encountered in 2016 and 2017 was still Python 2.7, with no migration plan, in production systems that generated real revenue.
Python 2 is dead now. The official CPython team stopped releasing security patches after January 2020. If you are still running Python 2 in production in 2024, you are running unpatched code with known security vulnerabilities. The migration is overdue.
The lesson from the Python 2/3 saga is that breaking changes, even correct ones, carry an enormous community cost. Language designers at other projects cite Python's migration as a cautionary tale. Python 3.12 and 3.13 added significant performance improvements specifically because the core team knew the language needed to demonstrate continued momentum after the disruption.
CPython, PyPy, and Other Implementations
When developers say "Python," they mean CPython, the reference implementation written in C by Guido van Rossum and the core team. CPython interprets Python bytecode on a stack-based virtual machine. It is what you install from python.org.
PyPy is an alternative Python implementation with a JIT (Just-In-Time) compiler. PyPy analyzes running code, identifies hot loops, and compiles them to machine code at runtime. For CPU-bound Python code, PyPy routinely runs 5 to 10 times faster than CPython. For I/O-bound applications, the speedup is smaller because the bottleneck is waiting for network or disk, not executing Python bytecode.
PyPy has full compatibility with Python 3.10 as of 2024. The main limitation is that C extensions (NumPy, Pandas, PyTorch) run through a compatibility layer that reduces PyPy's performance advantage for scientific computing workloads. If your bottleneck is pure Python business logic rather than numerical computation, PyPy is worth benchmarking.
Jython runs Python on the JVM. It provides access to Java libraries from Python code and can be embedded in Java applications. It lags behind CPython versions significantly and sees limited production use.
MicroPython and CircuitPython are Python implementations for microcontrollers. They run on hardware with kilobytes of RAM. If you write Python to control hardware, these are the runtimes you target.
The GIL (Global Interpreter Lock) exists in CPython specifically, not in all Python implementations. Jython and IronPython (Python on .NET) do not have the GIL. PyPy has its own GIL. Removing the GIL from CPython has been the subject of active work since 2022, and Python 3.13 ships an experimental free-threaded mode you can enable at compile time.
Type Hints and Static Analysis
Python's dynamic typing moved fast and made development feel frictionless. It also produced runtime errors that type checking at compile time would have caught before code shipped. Starting with Python 3.5, the language added optional type annotations. Starting with Python 3.9 and 3.10, the syntax became significantly cleaner.
from typing import TypeAlias, Protocol, runtime_checkable
from collections.abc import Iterator, AsyncIterator, Callable
from dataclasses import dataclass, field
UserID: TypeAlias = int
ProductID: TypeAlias = int
@runtime_checkable
class Repository(Protocol):
def get(self, id: int) -> dict | None: ...
def save(self, data: dict) -> dict: ...
def delete(self, id: int) -> bool: ...
@dataclass
class PaginatedResult[T]:
items: list[T]
total: int
page: int
per_page: int
pages: int = field(init=False)
def __post_init__(self):
self.pages = -(-self.total // self.per_page)
@property
def has_next(self) -> bool:
return self.page < self.pages
@property
def has_prev(self) -> bool:
return self.page > 1
def paginate[T](
items: list[T],
page: int,
per_page: int
) -> PaginatedResult[T]:
total = len(items)
start = (page - 1) * per_page
end = start + per_page
return PaginatedResult(
items=items[start:end],
total=total,
page=page,
per_page=per_page
)
def retry[T](
func: Callable[[], T],
max_attempts: int = 3,
exceptions: tuple[type[Exception], ...] = (Exception,)
) -> T:
last_exception: Exception | None = None
for attempt in range(max_attempts):
try:
return func()
except exceptions as e:
last_exception = e
if attempt == max_attempts - 1:
raise
raise last_exception
TypeAlias creates named type aliases. UserID: TypeAlias = int does not create a new type at runtime. It tells your type checker that UserID should be treated as int throughout the codebase. For documentation and readability, named aliases make function signatures far more informative.
The Protocol class defines structural subtypes. Any class that implements the methods defined in Repository satisfies the Repository protocol, without requiring inheritance. This is duck typing with static verification. @runtime_checkable adds isinstance() support for the protocol.
Generic syntax in Python 3.12+ uses [T] directly on functions and classes. Before 3.12, generics required TypeVar and separate declarations. The new syntax is cleaner and reads like other modern statically typed languages.
mypy and pyright are the two main static type checkers for Python. Pyright ships inside Pylance, the VS Code Python extension. Both read your type annotations and surface type errors before runtime. Running a type checker on a large Python codebase catches a category of bugs that tests often miss.
For new projects, write type annotations from the start. Retrofitting annotations onto an existing untyped codebase is painful. The upfront cost is low. The long-term benefit in maintainability and tooling support is substantial.
Concurrency: asyncio, Threading, and Multiprocessing
Python gives you three concurrency models. Choosing the wrong one for your workload costs performance you never recover.
asyncio handles I/O concurrency in a single thread. One event loop runs on one CPU core and switches between coroutines at await points. If a coroutine is waiting for a database response, the event loop runs other coroutines. No OS threads involved. No thread switching overhead. This is the right model for web servers, API services, Discord bots, and anything that spends most of its time waiting for network or disk.
import asyncio
import aiohttp
import time
from typing import NamedTuple
class FetchResult(NamedTuple):
url: str
status: int
elapsed: float
content_length: int
async def fetch_url(session: aiohttp.ClientSession, url: str) -> FetchResult:
start = time.monotonic()
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as response:
content = await response.read()
elapsed = time.monotonic() - start
return FetchResult(
url=url,
status=response.status,
elapsed=elapsed,
content_length=len(content)
)
async def fetch_all(urls: list[str], concurrency: int = 10) -> list[FetchResult]:
semaphore = asyncio.Semaphore(concurrency)
results = []
async def fetch_with_semaphore(session: aiohttp.ClientSession, url: str) -> FetchResult:
async with semaphore:
return await fetch_url(session, url)
async with aiohttp.ClientSession() as session:
tasks = [fetch_with_semaphore(session, url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
return [r for r in results if isinstance(r, FetchResult)]
async def main():
urls = [
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/1',
'https://httpbin.org/delay/1',
'https://httpbin.org/status/200',
'https://httpbin.org/status/404',
]
start = time.monotonic()
results = await fetch_all(urls, concurrency=5)
total = time.monotonic() - start
for result in results:
print(f"{result.status} {result.url} ({result.elapsed:.2f}s, {result.content_length} bytes)")
print(f"\nFetched {len(results)} URLs in {total:.2f}s")
asyncio.run(main())
asyncio.Semaphore(concurrency) limits concurrent requests to concurrency at a time. Without a semaphore, asyncio.gather() fires all requests simultaneously. At high URL counts, this exhausts file descriptors and overwhelms target servers. The semaphore acts as a rate limiter at the coroutine level.
return_exceptions=True in asyncio.gather() prevents one failed request from cancelling all other requests. Failed tasks return exception objects instead of raising. Filter them out after all tasks complete.
Threading suits CPU-bound work where C extensions release the GIL, and for code that must use synchronous blocking libraries:
import concurrent.futures
import hashlib
import os
def hash_file(filepath: str) -> tuple[str, str]:
sha256 = hashlib.sha256()
with open(filepath, 'rb') as f:
for chunk in iter(lambda: f.read(65536), b''):
sha256.update(chunk)
return filepath, sha256.hexdigest()
def hash_directory(directory: str, max_workers: int = 4) -> dict[str, str]:
files = [
os.path.join(root, filename)
for root, _, filenames in os.walk(directory)
for filename in filenames
]
results = {}
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {executor.submit(hash_file, f): f for f in files}
for future in concurrent.futures.as_completed(futures):
try:
filepath, digest = future.result()
results[filepath] = digest
except Exception as e:
print(f"Error hashing {futures[future]}: {e}")
return results
Multiprocessing bypasses the GIL entirely by spawning separate processes:
import multiprocessing
from multiprocessing import Pool
import numpy as np
from typing import Any
def process_chunk(args: tuple[np.ndarray, int]) -> dict[str, Any]:
data, chunk_id = args
return {
'chunk_id': chunk_id,
'mean': float(np.mean(data)),
'std': float(np.std(data)),
'min': float(np.min(data)),
'max': float(np.max(data)),
'size': len(data)
}
def parallel_analysis(data: np.ndarray, n_workers: int = None) -> list[dict]:
n_workers = n_workers or multiprocessing.cpu_count()
chunks = np.array_split(data, n_workers)
with Pool(processes=n_workers) as pool:
results = pool.map(process_chunk, [(chunk, i) for i, chunk in enumerate(chunks)])
return results
data = np.random.randn(10_000_000)
results = parallel_analysis(data)
for r in results:
print(f"Chunk {r['chunk_id']}: mean={r['mean']:.4f}, std={r['std']:.4f}, n={r['size']:,}")
Pool.map() distributes work across worker processes. Each process gets its own Python interpreter with its own GIL. For CPU-bound numerical work, this saturates all available CPU cores. The tradeoff is inter-process communication overhead, which makes multiprocessing inefficient for small tasks that complete faster than the cost of spawning processes.
The decision tree: use asyncio for I/O-bound concurrent work. Use threading for I/O-bound work with blocking libraries. Use multiprocessing for CPU-bound work that needs parallelism. When in doubt, profile first.
Should You Learn Python?
If you work in data science, machine learning, or AI, Python is not optional. The entire ecosystem runs on it. You will use NumPy, Pandas, PyTorch or TensorFlow, and Jupyter. Every major ML framework ships Python bindings first. This is where Python dominates without competition.
If you build backend services and APIs, Python is a strong choice. The async ecosystem has matured. FastAPI is genuinely excellent. The deployment story with Docker is straightforward. You get access to every library ever written for any domain.
If you want to automate infrastructure, process data, write CLI tools, or build internal tooling, Python gets you there faster than anything else. The "batteries included" standard library means you write real programs quickly.
If you build mobile apps or frontend web applications, Python is the wrong language for those specific domains.
I spent years ignoring Python because I already knew other languages and did not see the point of adding another. That was a mistake. Not because Python replaces the other languages. Because Python covers a surface area of software development that nothing else covers as well. The data science ecosystem alone justifies learning it. The fact that it also runs the backend of some of the world's largest web applications confirms that scale is not its enemy.
The Discord bot going from 3 servers to 100,000+ is a good story. But Instagram scaling from zero to 2 billion users on Python is the story that matters. Dropbox storing hundreds of millions of users' files in Python is the story that matters. Netflix processing real-time streaming analytics in Python is the story that matters.
You should learn Python. Everyone who told me that was right.
