Is a repository layer over sqlc over-engineering or necessary for scale? I’m building a notification engine in Go using sqlc for the DB layer. Do you just inject *db.Queries into your services, or do you find the abstraction of a repository layer worth the extra code?

I attempted to answer it there and the gist is correct. But I wrote it in a hurry so the example and the explanation could be better. Capturing it properly here.

Call it repository or whatever you want, the name doesn’t matter. The point is that your business logic should be oblivious to the persistence layer. Doesn’t matter if it’s sqlc, raw database/sql, or gorm. If your service functions call sqlc queries directly, your core logic is coupled to your database code. That makes it harder to test in isolation and harder to swap out later.

Put a small interface between your business code and your storage code. The business side defines what it needs, the storage side satisfies it, and they live in separate packages.

Say you’re building a service that manages books. Start with the domain type and the storage interface:

// bookstore/bookstore.go

type Book struct {
    ID    int64
    Title string
}

type BookStore interface {
    Get(ctx context.Context, id int64) (Book, error)
    Create(ctx context.Context, b Book) (int64, error)
}

The service depends only on that interface:

// bookstore/service.go

type Service struct {
    store BookStore
}

func NewService(s BookStore) *Service {
    return &Service{store: s}
}

func (s *Service) RegisterBook(
    ctx context.Context, title string) (Book, error) {

    b := Book{Title: title}
    id, err := s.store.Create(ctx, b)
    if err != nil {
        return Book{}, err
    }
    b.ID = id
    return b, nil
}

func (s *Service) GetBook(ctx context.Context, id int64) (Book, error) {
    return s.store.Get(ctx, id)
}

RegisterBook doesn’t know about SQL, sqlc, or Postgres. It builds a Book, asks the store to persist it, and gets an ID back.

The concrete implementation goes in a separate package. This is where sqlc-generated code would live:

// postgres/store.go

type Store struct{ db *sql.DB }

func NewStore(db *sql.DB) *Store { return &Store{db: db} }

func (s *Store) Get(ctx context.Context, id int64) (bookstore.Book, error) {
    // sqlc query or raw sql, doesn't matter
    // ...
}

func (s *Store) Create(
    ctx context.Context, b bookstore.Book) (int64, error) {
    // INSERT INTO books (title) VALUES ($1) RETURNING id
    // ...
}

Wire it up at startup:

// cmd/main.go

store := postgres.NewStore(db)
svc := bookstore.NewService(store)

In tests, swap in a fake that satisfies the same interface:

// bookstore/service_test.go

var _ BookStore = (*memStore)(nil)

type memStore struct {
    mu   sync.Mutex
    data map[int64]Book
    next int64
}

func (m *memStore) Get(
    ctx context.Context, id int64) (Book, error) {

    m.mu.Lock()
    defer m.mu.Unlock()
    b, ok := m.data[id]
    if !ok {
        return Book{}, fmt.Errorf("book %d not found", id)
    }
    return b, nil
}

func (m *memStore) Create(
    ctx context.Context, b Book) (int64, error) {

    m.mu.Lock()
    defer m.mu.Unlock()
    m.next++
    b.ID = m.next
    m.data[b.ID] = b
    return b.ID, nil
}

Now the test reads exactly like production code, minus Postgres:

// bookstore/service_test.go

func TestRegisterBook(t *testing.T) {
    store := &memStore{data: make(map[int64]Book)}
    svc := NewService(store)

    b, err := svc.RegisterBook(context.Background(), "DDIA")
    if err != nil {
        t.Fatal(err)
    }
    if b.ID == 0 {
        t.Fatal("expected non-zero ID")
    }
    if b.Title != "DDIA" {
        t.Fatalf("got title %q, want DDIA", b.Title)
    }
}

Same service code, no database needed. The test exercises RegisterBook without touching SQL. If the storage layer changes tomorrow, the service and its tests stay the same.

etcd is a distributed key-value store where the server and client communicate over gRPC. But if you’ve only ever used clientv3 and never peeked into the internals, you wouldn’t know that. You call resp, err := client.Put(ctx, "key", "value") and get back a *PutResponse. It feels like a regular Go library. The fact that gRPC and protobuf are involved is an implementation detail that the client wrapper keeps away from you.

I’ve been building a few gRPC services at work lately, and I keep running into the same question: what API do the users of my client library see? The server ships as a binary. The client ships as a Go package that other teams go get. If I hand them the raw generated gRPC stubs, they have to import my protobuf types, manage gRPC connections, configure TLS, and parse codes.NotFound from google.golang.org/grpc/status. That’s a lot of protocol plumbing for someone who just wants to consume my service.

This post walks through wrapping a generated gRPC client behind a higher level Go API, following the same pattern etcd uses. The idea is to give the user a wrapper client that abstracts out the generated client.

I’ll use a small in-memory KV store as the running example.

Layout

kv/
├── api/
│   ├── kv.proto           # service definition
│   ├── kv.pb.go           # generated message types
│   └── kv_grpc.pb.go      # generated client and server stubs
├── client/
│   └── client.go          # the wrapper (what users import)
├── server/
│   └── main.go            # the server binary
└── go.mod

api/ holds the proto and generated code. server/ is a binary you deploy. client/ is the library you ship. Other teams add it to their go.mod and never touch proto types directly.

Defining the service

The KV store has three RPCs: put, get, and delete.

// api/kv.proto
syntax = "proto3";
package kvpb;
option go_package = "example.com/kv/api";

service KV {
  rpc Put(PutRequest) returns (PutResponse);
  rpc Get(GetRequest) returns (GetResponse);
  rpc Delete(DeleteRequest) returns (DeleteResponse);
}

message PutRequest    { string key = 1; bytes value = 2; }
message PutResponse   {}
message GetRequest    { string key = 1; }
message GetResponse   { bytes value = 1; optional bool found = 2; }
message DeleteRequest { string key = 1; }
message DeleteResponse {}

GetResponse uses optional bool found because proto3 normally can’t distinguish “field is zero” from “field was never set.” The optional keyword generates a pointer in Go, which lets callers tell a missing key apart from an empty value.

Running protoc on this generates a client interface and a server stub. The client side looks like this:

// api/kv_grpc.pb.go (generated)
type KVClient interface {
    Put(ctx context.Context, in *PutRequest,
        opts ...grpc.CallOption) (*PutResponse, error)
    Get(ctx context.Context, in *GetRequest,
        opts ...grpc.CallOption) (*GetResponse, error)
    Delete(ctx context.Context, in *DeleteRequest,
        opts ...grpc.CallOption) (*DeleteResponse, error)
}

Every method takes a context.Context, a protobuf request struct, and variadic grpc.CallOptions, and returns a protobuf response plus an error. Anyone calling the service has to import protobuf types, construct request structs like &api.PutRequest{}, and understand gRPC call options, even for a simple “get this key” call.

The server implements the other side with an in-memory map. What we care about for the wrapper is that it returns a gRPC NOT_FOUND status when a key doesn’t exist. The wrapper translates that into a Go sentinel error. Here’s the server code:

// server/main.go
type server struct {
    kvpb.UnimplementedKVServer
    data map[string][]byte
}

func (s *server) Get(
    ctx context.Context, r *kvpb.GetRequest,
) (*kvpb.GetResponse, error) {
    v, ok := s.data[r.Key]
    if !ok {
        return nil, status.Errorf(
            codes.NotFound, "key %q", r.Key)
    }
    return &kvpb.GetResponse{
        Value: v, Found: proto.Bool(true),
    }, nil
}
// Put and Delete follow the same shape.

The server embeds UnimplementedKVServer, the standard gRPC pattern. It provides no-op implementations for all RPCs so the code compiles even before you’ve written the real logic. The Get method checks the map and returns codes.NotFound when the key isn’t there. This is the status code the wrapper will catch and turn into a Go error. I’ve elided Put and Delete since they follow the same structure.

Using the generated client directly

Without a wrapper, callers use the generated KVClient directly. Pay attention to the imports:

// example/main.go (raw usage without wrapper)
import (
    "context"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
    "example.com/kv/api"
)

// ...
conn, err := grpc.NewClient("localhost:9090",
    grpc.WithTransportCredentials(insecure.NewCredentials()))
// ...
kv := api.NewKVClient(conn)
_, err = kv.Put(ctx, &api.PutRequest{
    Key: "greeting", Value: []byte("hello"),
})

Three imports just to put a key. The caller manages the gRPC connection, constructs &api.PutRequest{} structs for every call, and has to parse gRPC status codes to check if a key exists. For internal code where everyone knows gRPC, this is fine. For a library you ship to other teams, it’s a lot of ceremony.

Calling the server with the wrapper

This is the API we actually want to give our users. Same sequence as before (put a key, get it back, handle a missing key) but without any gRPC or protobuf leaking through:

// example/main.go (with the wrapper)
import "example.com/kv/client"

// ...

c, err := client.New("localhost:9090")
// ...
defer c.Close()

err = c.Put(ctx, "greeting", []byte("hello"))

val, err := c.Get(ctx, "greeting")

_, err = c.Get(ctx, "missing")
if errors.Is(err, client.ErrNotFound) { ... }

One import instead of three. No gRPC or protobuf packages in sight. Put takes a string and a byte slice. Get returns []byte. Missing keys come back as client.ErrNotFound, checked with errors.Is like any other Go error. The caller doesn’t need to know that gRPC is involved at all.

The rest of this post builds the wrapper that turns the generated KVClient from the previous section into this API.

Building the wrapper

The client/ package is the only thing users import. It hides the generated api.KVClient behind a struct and re-exposes the same operations using plain Go types. The whole wrapper lives in a single file (client/client.go).

The wrapper starts with a sentinel error and a testable interface:

// client/client.go

var ErrNotFound = errors.New("key not found")

type KV interface {
    Put(ctx context.Context, key string, value []byte) error
    Get(ctx context.Context, key string) ([]byte, error)
    Delete(ctx context.Context, key string) error
}

ErrNotFound replaces the gRPC NOT_FOUND status code. Callers check it with errors.Is and never import google.golang.org/grpc/codes.

Client implements KV, and KV uses only standard Go types instead of protobuf or gRPC types. This is intentionally a producer-side interface: we define it in the same package as Client because we know the full set of operations the service supports and we want to offer a ready-made contract for consumers. Other packages that depend on your client can accept a KV in their function signatures and swap in a simple in-memory fake during tests without spinning up a gRPC server or importing any gRPC packages.

Then the struct and constructor:

type Client struct {
    conn *grpc.ClientConn
    kv   api.KVClient
}

func New(addr string, opts ...grpc.DialOption) (*Client, error) {
    if len(opts) == 0 {
        opts = []grpc.DialOption{
            grpc.WithTransportCredentials(insecure.NewCredentials()),
        }
    }
    conn, err := grpc.NewClient(addr, opts...)
    if err != nil {
        return nil, fmt.Errorf("connecting to %s: %v", addr, err)
    }
    return &Client{conn: conn, kv: api.NewKVClient(conn)}, nil
}

func (c *Client) Close() error { return c.conn.Close() }

Client holds the gRPC connection and the generated api.KVClient as unexported fields. Note that api.KVClient is an interface, not a concrete struct. The gRPC codegen doesn’t expose the actual client struct at all; you get back a KVClient interface from api.NewKVClient(conn). We store it as a regular field rather than embedding it. If you embedded the api.KVClient interface, all its methods like Put(ctx, *PutRequest, ...CallOption) would be promoted onto Client directly, and callers could bypass the wrapper to make raw gRPC calls.

New creates the gRPC connection and builds the generated client from it. The variadic grpc.DialOption lets callers pass custom TLS, keepalive, or interceptor config. If they pass nothing, the default is insecure credentials for local dev. The retries section below shows what a production setup looks like.

With the types in place, we can look at the wrapper methods. Get shows the pattern all three follow:

func (c *Client) Get(ctx context.Context, key string) ([]byte, error) {
    resp, err := c.kv.Get(ctx, &api.GetRequest{Key: key})
    if err != nil {
        if s, ok := status.FromError(err); ok &&
            s.Code() == codes.NotFound {
            return nil, ErrNotFound
        }
        return nil, fmt.Errorf(
            "getting key %s: %v", key, err)
    }
    return resp.Value, nil
}
// Put and Delete follow the same shape.

Each wrapper method follows the same pattern: take the caller’s Go arguments, build the protobuf request internally, call the generated client, and return plain Go types.

Pay attention to the error handling. When the server returns NOT_FOUND, we catch that gRPC status and convert it to our own ErrNotFound sentinel so callers can check it with errors.Is instead of parsing gRPC status codes themselves. For everything else, we wrap with %v instead of %w. If we used %w, callers could unwrap the error with errors.As and reach the underlying gRPC status types, which would re-couple them to gRPC internals and defeat the whole point of having a wrapper. I wrote about this tradeoff in Go errors: to wrap or not to wrap?.

Plugging in retries and metrics

Since the wrapper owns the grpc.NewClient call, it can bake in retries and observability without the caller knowing. gRPC interceptors work like HTTP middleware. They wrap every RPC with extra logic (logging, retries, metrics) without changing the handler code. You register them as dial options when creating the connection:

// client/client.go (production version of New)
func New(addr string, opts ...grpc.DialOption) (*Client, error) {
    defaults := []grpc.DialOption{
        grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})),
        grpc.WithChainUnaryInterceptor(
            grpc_retry.UnaryClientInterceptor(
                grpc_retry.WithMax(3),
                grpc_retry.WithBackoff(
                    grpc_retry.BackoffExponential(100*time.Millisecond),
                ),
            ),
            grpcprom.UnaryClientInterceptor,
        ),
    }
    opts = append(defaults, opts...)
    // ... rest is the same
}

grpc_retry from go-grpc-middleware retries failed RPCs with exponential backoff. grpcprom records latency histograms and error rates. Same client.New, same c.Put, but now with retries and metrics baked in. Callers who need to override the defaults can pass their own dial options. This is useful in tests where you might want insecure credentials or no retries.

Try it yourself

The full code is on GitHub. Install the server and run the example:

go install github.com/rednafi/examples/wrapping-grpc-client/server@latest
server &

go install github.com/rednafi/examples/wrapping-grpc-client/example@latest
example

Running the example will return:

put greeting=hello
get greeting=hello
get missing: not found (expected)
deleted greeting
get greeting after delete: not found (expected)

Or add the client library to your own project:

go get github.com/rednafi/examples/wrapping-grpc-client/client@latest

I was mainly looking for pointers on how to organize protobuf definitions, wire up server-side metrics and interceptors, and build ergonomic client wrappers. The default answer here is often “go read the Docker or Kubernetes codebase.” But both of those are pretty huge and take forever to get accustomed to.

Then I found etcd. It’s used by Kubernetes’ control plane for storing configs in a consistent manner. It exposes a small set of well-defined gRPC endpoints to interact with the storage layer. The core services are defined in a single rpc.proto file:

service KV {
  rpc Range(RangeRequest) returns (RangeResponse);
  rpc Put(PutRequest) returns (PutResponse);
  rpc DeleteRange(DeleteRangeRequest) returns (DeleteRangeResponse);
  rpc Txn(TxnRequest) returns (TxnResponse);
  rpc Compact(CompactionRequest) returns (CompactionResponse);
}

// ...

The full file also defines Watch, Lease, Cluster, Maintenance, and Auth services. Grokking that file and the surrounding api directory is a good way to learn how to organize your protobufs and generated code. Some other things I picked up:

• Proto definitions live under api/, separated into subpackages like etcdserverpb, mvccpb, authpb. Generated Go code lives alongside the proto files.

• The RPC handler implementations live under server/etcdserver/api/v3rpc. key.go implements the KV service (Range, Put, DeleteRange, Txn, Compact), and the other services follow the same pattern in watch.go, lease.go, member.go, maintenance.go, auth.go.

• grpc.go shows how to assemble a gRPC server with chained unary and stream interceptors using go-grpc-middleware.

• Server-side Prometheus metrics are wired in grpc.go via grpc_prometheus.ServerMetrics interceptors. It optionally enables latency histograms when the metric type is extensive.

• metrics.go defines custom Prometheus counters and histograms on top of the standard gRPC ones, things like etcd_network_client_grpc_sent_bytes_total and watch stream durations.

• interceptor.go handles logging. newLogUnaryInterceptor logs request/response sizes at warn level when latency exceeds a threshold.

• The client has no built-in metrics. The clientv3 README says you can wire up go-grpc-prometheus yourself, but the library doesn’t do it for you.

• retry_interceptor.go implements client-side retry with backoff, safe retry classification for read-only vs mutation RPCs, and auth token refresh on failure.

• The clientv3 package wraps the generated gRPC client behind a nicer Go API. Good reference if you’re building an ergonomic client on top of raw protobuf types.

• If you’re a distributed systems nerd, etcd uses Raft for consensus. That part of the codebase is its own rabbit hole.

This has become my go-to whenever I’m wiring up another gRPC service at work. I’ve gotten comfortable enough with it over the last few weeks that I can point people to specific files when we need to make decisions.

• When to fmt.Errorf("doing X: %w", err)
• When to use %v instead of %w
• When to just return err

There’s no consensus, and the answer changes depending on the kind of application you’re writing. The Go 1.13 blog already covers the mechanics and offers some guidance, but I wanted to collect more evidence of what people are actually doing in the open and share what’s worked for me.

The problem with bare errors

Here’s a function that places an order by calling into a few different packages:

func placeOrder(ctx context.Context, req OrderReq) error {
    user, err := users.Get(ctx, req.UserID)
    if err != nil {
        return err
    }
    err = inventory.Reserve(ctx, req.ItemID, req.Qty)
    if err != nil {
        return err
    }
    err = payments.Charge(ctx, user.PaymentID, req.Total)
    if err != nil {
        return err
    }
    return saveOrder(ctx, user.ID, req.ItemID)
}

All four calls can fail with connection refused. When one of them does, your log says:

connection refused

Which call? No idea. You grep the codebase, add temporary logging, narrow it down. In a service with dozens of dependencies, debugging this trail of errors can turn into a huge time sink.

One obvious fix is to wrap the error at every return site:

user, err := users.Get(ctx, req.UserID)
if err != nil {
    return fmt.Errorf("getting user %s: %w", req.UserID, err)
}
err = inventory.Reserve(ctx, req.ItemID, req.Qty)
if err != nil {
    return fmt.Errorf("reserving stock for %s: %w", req.ItemID, err)
}

Now the log says:

reserving stock for item-123: connection refused

That tells you exactly which call failed and which item it was for.

The case for wrapping at every return site

Dave Cheney advocated for this in his 2016 talk Don’t just check errors. His pkg/errors library introduced errors.Wrap, which adds a message and a stack trace at the point where the error occurs. The idea is that each function knows what operation it was attempting, and that context is lost if you don’t capture it immediately.

CockroachDB takes this further. They use cockroachdb/errors, a drop-in replacement for the stdlib errors package that captures a stack trace at every wrap site:

// cockroachdb style: stack trace at every wrap
if err := r.validateCmd(ctx, cmd); err != nil {
    return errors.Wrap(err, "validating command")
}
if err := r.stage(ctx, cmd); err != nil {
    return errors.Wrap(err, "staging command")
}

The Terraform AWS provider does the same thing with fmt.Errorf("...: %w", err) at every layer. Their contributor guidelines mandate a consistent format for all resource operations:

// terraform-provider-aws style
output, err := conn.CreateVpc(ctx, input)
if err != nil {
    return fmt.Errorf("creating EC2 VPC: %w", err)
}

d.SetId(aws.ToString(output.Vpc.VpcId))

if _, err := WaitVPCAvailable(ctx, conn, d.Id()); err != nil {
    return fmt.Errorf(
        "waiting for EC2 VPC (%s) available: %w",
        d.Id(), err,
    )
}

The wrapcheck linter codifies this as a rule. It doesn’t flag every bare return err, only errors that originated from a different package:

func placeOrder(ctx context.Context, req OrderReq) error {
    // users.Get is in another package: wrapcheck flags
    user, err := users.Get(ctx, req.UserID)
    if err != nil {
        return err // not wrapped: linter warning
    }

    // validate is in the same package: wrapcheck allows
    err = validate(req)
    if err != nil {
        return err // fine, same package
    }
    // ...
}

The reasoning is that when an error crosses a package boundary, the receiving code is the last place that knows what it was trying to do. Within a package, the caller already has that context.

For many cases, wrapping everything is the right default:

The risk of overwrapping, especially in my private code, is much lower than the risk of underwrapping when the service crashes and you get io.EOF.

– Peter Bourgon on Go Time #91

But wrapping has costs that only show up as the codebase grows.

The cost of overwrapping

Messages pile up

When every layer wraps, your error messages become nested chains:

placing order: reserving stock for item-123:
    checking warehouse: querying database:
    connection refused

Four layers of context for one connection refused. The middle layers (checking warehouse and querying database) don’t add a warehouse ID or a query. They just restate the call chain.

It also makes the error string fragile. It changes whenever someone renames an intermediate function or refactors the call chain. If you had an alert matching on checking warehouse: querying database: connection refused, it breaks the moment someone renames checkWarehouse to checkStock. The same root cause (connection refused) wrapped through different code paths produces different error strings, making it hard to aggregate them in your logging dashboard.

Jay Conrod’s error handling guidelines address this:

Each function is responsible for including its own values in the error message, except for arguments passed to the function that returned the wrapped error.

In other words, if os.Open already puts the file path in its error, your wrapper shouldn’t add the path again:

// redundant: the path appears twice
return fmt.Errorf("opening %s: %w", path, err)
// open /etc/app.yaml: opening /etc/app.yaml: permission denied

// better: add what you were doing, not what Open already said
return fmt.Errorf("reading config: %w", err)
// reading config: open /etc/app.yaml: permission denied

The Google Go Style Guide says the same:

When adding information to errors, avoid redundant information that the underlying error already provides.

You should still wrap, but only when you’re adding information - a user ID, an item ID, the name of the external service you were calling.

%w creates contracts you didn’t mean to

%w in fmt.Errorf creates an error chain that callers can traverse with errors.Is and errors.As. That means the wrapped error becomes part of your function’s API surface.

The Go 1.13 blog uses sql.ErrNoRows to illustrate this. Say your LookupUser function calls database/sql internally:

func LookupUser(ctx context.Context, id string) (*User, error) {
    row := db.QueryRowContext(ctx, "SELECT ...", id)
    var u User
    if err := row.Scan(&u.Name, &u.Email); err != nil {
        return nil, fmt.Errorf(
            "looking up user %s: %w", id, err,
        )
    }
    return &u, nil
}

Because of %w, callers can now do errors.Is(err, sql.ErrNoRows) to check whether the user wasn’t found. That works until you switch from database/sql to an ORM, or put a cache in front of the query. The callers matching on sql.ErrNoRows silently break.

The Go 1.13 blog is explicit about this:

Wrapping an error makes that error part of your API. If you don’t want to commit to supporting that error as part of your API in the future, you shouldn’t wrap the error.

The Error Values FAQ makes the same point:

Callers can depend on the type and value of the error you’re wrapping, so changing that error can now break them. […] At that point, you must always return sql.ErrTxDone if you don’t want to break your clients, even if you switch to a different database package.

Same thing with typed errors. If your repository wraps a pgconn.PgError with %w, callers can unwrap through to the Postgres error code:

if pgErr, ok := errors.AsType[*pgconn.PgError](err); ok {
    log.Println(pgErr.Code) // e.g. "23505" (unique violation)
}

When you migrate to MySQL or put a cache in front of the database, those callers silently break.

The Google Go Style Guide notes that %w is appropriate when your package’s API guarantees that certain underlying errors can be unwrapped and checked by callers. If you don’t want to make that guarantee, use %v.

%v as the conservative default

%v adds the same context text (the human reading the log sees the identical message) but severs the error chain. No caller can errors.Is or errors.As through it:

// %w: callers can errors.Is(err, sql.ErrNoRows)
return fmt.Errorf("getting user %s: %w", id, err)

// %v: same message text, but the chain is severed
return fmt.Errorf("getting user %s: %v", id, err)

Both produce the same log output. But with %v, you’re free to swap the database later without breaking callers who were depending on the inner error type.

At system boundaries, the Google Go Style Guide recommends translating rather than wrapping:

At points where your system interacts with external systems like RPC, IPC, or storage, it’s often better to translate domain-specific errors into a standardized error space (e.g., gRPC status codes) rather than simply wrapping the raw underlying error with %w.

Say your repository layer talks to Postgres via pgx. Wrapping with %w exposes pgx errors to callers:

func (r *UserRepo) Get(ctx context.Context, id string) (*User, error) {
    row := r.db.QueryRow(ctx, "SELECT ...", id)
    if err := row.Scan(&u.Name, &u.Email); err != nil {
        return nil, fmt.Errorf("getting user %s: %w", id, err)
    }
    return &u, nil
}

Now any caller can errors.Is(err, pgx.ErrNoRows), tying them to your database driver. Translating means mapping the storage error into your own domain before it crosses the boundary:

var ErrNotFound = errors.New("not found")

func (r *UserRepo) Get(ctx context.Context, id string) (*User, error) {
    row := r.db.QueryRow(ctx, "SELECT ...", id)
    if err := row.Scan(&u.Name, &u.Email); err != nil {
        if errors.Is(err, pgx.ErrNoRows) {
            return nil, ErrNotFound
        }
        return nil, fmt.Errorf("getting user %s: %v", id, err)
    }
    return &u, nil
}

Callers check errors.Is(err, ErrNotFound) - which is yours - instead of errors.Is(err, pgx.ErrNoRows). When you swap from Postgres to MySQL, callers don’t break. And at system boundaries, consider translating entirely instead of wrapping.

How the stdlib handles errors

The standard library also uses sentinel errors and custom error types alongside %w and %v.

Packages like io define sentinel errors - package-level variables that callers check with errors.Is. The io package defines EOF and returns it from Read when there’s no more data:

// definition
var EOF = errors.New("EOF")

// inside a Reader implementation
func (r *myReader) Read(p []byte) (int, error) {
    if r.pos >= len(r.data) {
        return 0, io.EOF
    }
    // ...
}

A caller uses the sentinel to distinguish “end of input” from a real failure:

n, err := reader.Read(buf)
if errors.Is(err, io.EOF) {
    // done reading, not an error
    break
}
if err != nil {
    return err
}

Sentinels work when the caller only needs to know which failure occurred. When callers need structured metadata - not just identity - the stdlib uses custom error types. os.Open defines a *fs.PathError struct and returns it with the operation name, file path, and underlying syscall error as struct fields:

// definition in the fs package
type PathError struct {
    Op   string // "open", "read", "write"
    Path string // the file path
    Err  error  // the underlying syscall error
}

func (e *PathError) Unwrap() error { return e.Err }

// inside os.Open
func Open(name string) (*File, error) {
    // ...
    return nil, &PathError{Op: "open", Path: name, Err: err}
}

Because PathError implements Unwrap(), errors.Is(err, fs.ErrNotExist) works through the chain. But unlike fmt.Errorf wrapping, the context is in typed struct fields. A caller can extract those fields to decide what to do:

f, err := os.Open("/etc/app.yaml")
if err != nil {
    if pathErr, ok := errors.AsType[*fs.PathError](err); ok {
        // pathErr.Op is "open", pathErr.Path is "/etc/app.yaml"
        // pathErr.Err is the syscall error (e.g. ENOENT)
        log.Printf(
            "%s failed on %s: %v",
            pathErr.Op, pathErr.Path, pathErr.Err,
        )
    }
    return err
}

net.OpError follows the same pattern with Op, Net, Source, Addr, and Err fields. The package controls exactly what’s exposed via Unwrap(), and callers get structured metadata they can act on programmatically.

The stdlib also uses fmt.Errorf with both %w and %v, and the database/sql package shows why the choice matters. Rows.Scan wraps scanner errors with %w:

return fmt.Errorf(
    `sql: Scan error on column index %d, name %q: %w`,
    i, rs.rowsi.Columns()[i], err,
)

Before Go 1.16, Rows.Scan used %v here, which severed the chain. Custom Scanner implementations returning sentinel errors couldn’t be inspected with errors.Is by callers. Issue #38099 fixed this by switching to %w. But in the same package, internal type conversion errors use %v because the underlying strconv parse error is an implementation detail callers don’t need to inspect:

return fmt.Errorf(
    "converting driver.Value type %T (%q) to a %s: %v",
    src, s, dv.Kind(), err,
)

The database/sql migration from %v to %w was safe because it only exposed more to callers. Going the other direction would break callers who started depending on errors.Is.

Kubernetes went through a similar migration. They historically used %v for most wrapping, which meant errors.As couldn’t traverse the chain. Issue #123234 tracked the codebase- wide migration from %v to %w, acknowledging that %v may still be preferred in some places “to abstract the implementation details” but that such cases should be rare.

For most application code, fmt.Errorf with %w or %v is enough. Custom error types like PathError make more sense in libraries and shared packages where callers need structured metadata. But wrapping isn’t the only way to attach context to an error.

Structured logging as an alternative to wrapping

Dave Cheney is the person who created pkg/errors and popularized error wrapping in Go. He eventually walked away from his own advice. In 2021, when looking for new maintainers for pkg/errors, he wrote:

I no longer use this package, in fact I no longer wrap errors.

– Dave Cheney on pkg/errors #245

His reasoning was that structured logging can carry the debugging context that wrapping was meant to provide. Compare the two approaches. With wrapping, you bake the context into the error string:

err = inventory.Reserve(ctx, req.ItemID, req.Qty)
if err != nil {
    return fmt.Errorf(
        "reserving stock for %s: %w", req.ItemID, err,
    )
}

The log line looks like:

reserving stock for item-123: connection refused

With structured logging, you keep the error value clean and attach the context as separate key-value fields:

err = inventory.Reserve(ctx, req.ItemID, req.Qty)
if err != nil {
    slog.Error("reserve stock failed",
        "item_id", req.ItemID,
        "err", err,
    )
    return err
}

The log line looks like:

level=ERROR msg="reserve stock failed"
    item_id=item-123 err="connection refused"

The same information is there, but in structured fields that your logging dashboard can index, filter, and aggregate on. The error value itself stays as connection refused without a chain of prefixes.

The tradeoff is that structured logging requires a logging pipeline that can query on fields. If all you have is grep on a log file, the wrapping version is easier to work with.

How wrapping changes by application type

So how do you actually decide? It depends on what you’re building. Marcel van Lohuizen from the Go team described his own approach:

I do and don’t… If I wanna have context, I wrap it. If I create a new error, I wrap it. But sometimes you’re not really adding too much information, and then I don’t. So it depends on the situation.

– Marcel van Lohuizen on Go Time #91

Libraries

Be conservative. The Google style guide applies most directly here because you’re shipping an API contract. Use %v by default so you don’t accidentally expose implementation details. Use %w only when you intentionally want callers to inspect the inner error, and document that you’re doing so.

A library that wraps with %w ties its callers to its dependencies. If v2 switches from pgx to database/sql, every caller doing errors.Is(err, pgconn.something) breaks. Use %v by default, and define your own sentinels when callers need to branch on the error:

var ErrNotFound = errors.New("item not found")

func (c *Client) Fetch(ctx context.Context, id string) (*Item, error) {
    resp, err := c.http.Get(ctx, c.url+"/items/"+id)
    if err != nil {
        if isNotFound(err) {
            return nil, ErrNotFound
        }
        return nil, fmt.Errorf("fetching item %s: %v", id, err)
    }
    // ...
}

Callers check errors.Is(err, ErrNotFound) - which is yours - without being coupled to your HTTP client. Same pattern as the UserRepo translation example earlier.

CLI tools

Wrap freely with %w. The call stack is shallow, the error message is the user-facing output, and nobody is calling errors.Is on your CLI’s errors. Maximum context helps the human reading the terminal:

func run() error {
    cfg, err := loadConfig(cfgPath)
    if err != nil {
        return fmt.Errorf("loading config %s: %w", cfgPath, err)
    }
    conn, err := connect(cfg.DatabaseURL)
    if err != nil {
        return fmt.Errorf("connecting to database: %w", err)
    }
    return migrate(conn)
}

The user sees:

loading config /etc/app.yaml:
    open /etc/app.yaml: permission denied

Services

In my experience, services are where it’s the hardest to give a formulaic answer to this. You have structured logging and distributed tracing, but you also have deep call stacks and many dependencies.

The approach I’ve landed on: wrap at package boundaries with context about what you were trying to do. Use %w within your own codebase where callers should be able to inspect the inner error. Use %v when the error crosses a system boundary (RPCs, database calls, third-party APIs). Skip wrapping for same-package calls.

Here’s the placeOrder function from the beginning, rewritten:

func placeOrder(ctx context.Context, req OrderReq) error {
    user, err := users.Get(ctx, req.UserID) // (1)
    if err != nil {
        return fmt.Errorf("getting user %s: %w", req.UserID, err)
    }
    err = inventory.Reserve(ctx, req.ItemID, req.Qty) // (2)
    if err != nil {
        return fmt.Errorf("reserving stock for %s: %w", req.ItemID, err)
    }
    err = payments.Charge(ctx, user.PaymentID, req.Total) // (3)
    if err != nil {
        return fmt.Errorf("charging payment: %w", err)
    }
    return saveOrder(ctx, user.ID, req.ItemID) // (4)
}

• (1) users.Get is in another package - wrap with the user ID
• (2) inventory.Reserve is in another package - wrap with the item ID
• (3) payments.Charge is in another package - wrap with the operation name
• (4) internal helper in the same package - bare return is enough

At the handler, use %v to translate into the external domain without exposing internals:

func handlePlaceOrder(
    ctx context.Context, req *pb.OrderReq,
) (*pb.OrderResp, error) {
    err := placeOrder(ctx, fromProto(req))
    if err != nil {
        slog.Error("placing order",
            "user_id", req.UserId,
            "item_id", req.ItemId,
            "err", err,
        )
        // %v: context for humans, no chain for callers
        return nil, status.Errorf(codes.Internal, "placing order: %v", err)
    }
    return &pb.OrderResp{}, nil
}

The handler logs the full error with request context for debugging, then returns a gRPC status with %v so the caller gets a useful message without being able to errors.Is through to your database driver.

Where I’ve landed

There’s no consensus on how much to wrap, and I don’t think there needs to be. Here’s what I do:

• Within a package, bare return err. The caller already has context.
• At package boundaries, fmt.Errorf("doing X: %w", err) with identifying info (user IDs, item IDs, file paths). The wrapcheck linter can enforce this automatically. Only wrap when you’re adding information the inner error doesn’t already carry.
• At system boundaries (RPCs, database calls, third-party APIs), translate rather than wrap. Map implementation errors into your own sentinel errors or custom error types so callers depend on your package, not your dependencies. Use %v for the fallback path.
• In libraries, %v by default. Own sentinels (ErrNotFound, ErrConflict) for cases callers need to inspect. %w only when you intentionally want callers to unwrap, and document that you’re doing so.
• In CLIs, %w everywhere. The error message is the user-facing output.
• In services, all of the above plus slog at the handler level for request-scoped context, so the error value doesn’t need to carry all of that.

var (
    mu      sync.Mutex
    counter int
)

mu.Lock()
counter++
mu.Unlock()

This works, but nothing enforces it. The compiler won’t stop you from accessing counter without holding the lock. Forget to lock in one spot and you have a data race. One way to make this safer is to bundle the value and its mutex into a small generic wrapper that only exposes locked access through methods:

type Locked[T any] struct {
    mu sync.Mutex
    v  T
}

func NewLocked[T any](initial T) *Locked[T] {
    return &Locked[T]{v: initial}
}

func (l *Locked[T]) Get() T {
    l.mu.Lock()
    defer l.mu.Unlock()
    return l.v
}

func (l *Locked[T]) Set(v T) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.v = v
}

You keep mu and v unexported, pass around *Locked[T], and callers use Get to read and Set to write:

counter := NewLocked(0)

counter.Set(42)
fmt.Println(counter.Get()) // 42

Now callers can’t touch the underlying value without going through the lock. This doesn’t prevent misuse within the same package, but it makes unprotected access from other packages impossible.

This works fine when you’re replacing the value wholesale - just call counter.Set(42) and move on. But when your mutation depends on the current value, Get and Set can race against each other.

The problem with Set

Say you want to increment the counter instead of replacing it. You’d have to do:

v := counter.Get()
v++
counter.Set(v)

Each individual call is safe - Get holds the lock while reading, Set holds it while writing. But the three calls together aren’t atomic. Between Get and Set, another goroutine can modify the value, and your increment overwrites theirs. That’s the classic lost-update bug.

It gets worse with compound state. Say the wrapper holds a struct:

type State struct {
    Count int
    Name  string
}

state := NewLocked(State{})

And you want to conditionally update both fields:

s := state.Get()
if s.Count < 10 {
    s.Count++
    s.Name = fmt.Sprintf("item-%d", s.Count)
}
state.Set(s)

Same problem. Get returns a copy, you mutate the copy, then Set writes it back. If another goroutine modified state between those two calls, your write clobbers it.

func TestSetValue(t *testing.T) {
    counter := NewLocked(0)

    var wg sync.WaitGroup
    for range 10 {
        wg.Go(func() {
            for range 1000 {
                v := counter.Get()
                v++
                counter.SetValue(v)
            }
        })
    }

    wg.Wait()

    got := counter.Get()
    if got != 10000 {
        t.Errorf("got %d, want 10000 (lost %d updates)", got, 10000-got)
    }
}

=== RUN   TestSetValue
    locked_test.go:30: got 1855, want 10000 (lost 8145 updates)
--- FAIL: TestSetValue (0.02s)

Take a function instead

Instead of taking a value, have Set take a function:

func (l *Locked[T]) Set(f func(*T)) {
    l.mu.Lock()
    defer l.mu.Unlock()
    f(&l.v)
}

Now the counter increment becomes:

counter.Set(func(v *int) {
    *v++
})

And the compound mutation:

state.Set(func(s *State) {
    if s.Count < 10 {
        s.Count++
        s.Name = fmt.Sprintf("item-%d", s.Count)
    }
})

The lock is held for the entire closure. There’s no gap between reading and writing, so no other goroutine can interfere. Both fields update together or not at all.

The function takes a pointer to T rather than a value of T for two reasons. First, it lets you mutate the state in place instead of working on a copy. Second, if T is a large struct, passing a pointer avoids copying the whole thing into the closure on every call.

The stdlib already does this

Go’s database/sql package has an internal withLock helper that follows the same pattern:

// withLock runs while holding lk.
func withLock(lk sync.Locker, fn func()) {
    lk.Lock()
    defer lk.Unlock() // in case fn panics
    fn()
}

It’s used throughout database/sql to serialize access to the underlying driver connection. For example, when pinging a connection:

if pinger, ok := dc.ci.(driver.Pinger); ok {
    withLock(dc, func() {
        err = pinger.Ping(ctx)
    })
}

Or when preparing a statement:

withLock(dc, func() {
    si, err = ctxDriverPrepare(ctx, dc.ci, query)
})

Or committing a transaction:

withLock(tx.dc, func() {
    err = tx.txi.Commit()
})

There are about 18 call sites in sql.go alone. In those snippets, dc is a *driverConn - the struct that wraps a database driver connection. It embeds sync.Mutex directly, so it satisfies sync.Locker and can be passed straight to withLock.

withLock(rs.closemu.RLocker(), func() {
    doClose, ok = rs.nextLocked()
})

The proposal to add this to sync

In 2021, twmb filed proposal #49563 to add a Mutex.Locked(func()) method to the standard library:

func (m *Mutex) Locked(fn func()) {
    m.Lock()
    defer m.Unlock()
    fn()
}

The idea was that if sync.Mutex had this method natively, you wouldn’t need to write a wrapper at all for simple cases - you’d just call mu.Locked(fn) directly. It also eliminates forgotten unlocks and guards against panics leaving the mutex locked. esote pointed out that database/sql already had an internal version of this - the same withLock helper we saw earlier.

zephyrtronium raised the sync.Locker point:

I think there are advantages to making this a function that takes a Locker rather than a method on Mutex. This would allow using it with either end of an RWMutex, or another custom Locker.

– zephyrtronium on #49563

rsc declined it on philosophical grounds:

In general we try not to have two different ways to do something, and for better or worse we have the current idioms.

– rsc on #49563

The more interesting pushback came from bcmills, who argued the proposal didn’t go far enough. With generics arriving, he wanted something that also prevents unguarded access to the protected data, not just forgotten unlocks:

Now that we have generics on the way, I would rather see us move in a direction that also eliminates unlocked-access bugs, not just incrementally update Mutex for forgotten-defer bugs.

– bcmills on #49563

He sketched out what that could look like:

type Synchronized[T any] struct {
    mu  Mutex
    val T
}

func (s *Synchronized[T]) Do(fn func(*T)) {
    s.mu.Lock()
    defer s.mu.Unlock()
    fn(&s.val)
}

This is essentially the Locked[T] wrapper from the beginning of this post. The proposal was declined, but bcmills’ suggestion is the direction the community ended up going anyway-just outside the standard library.

Tailscale’s MutexValue

Tailscale’s syncs package has a MutexValue[T] type that follows this direction:

type MutexValue[T any] struct {
    mu sync.Mutex
    v  T
}

func (m *MutexValue[T]) WithLock(f func(p *T)) {
    m.mu.Lock()
    defer m.mu.Unlock()
    f(&m.v)
}

func (m *MutexValue[T]) Load() T {
    m.mu.Lock()
    defer m.mu.Unlock()
    return m.v
}

func (m *MutexValue[T]) Store(v T) {
    m.mu.Lock()
    defer m.mu.Unlock()
    m.v = v
}

They provide both Store for simple replacements and WithLock for compound mutations. When you need to read-modify-write, you go through WithLock so the lock covers the whole operation.

When a plain Set is fine

If T is small and you only ever replace the whole value without reading it first, a plain Set works. A boolean flag that gets toggled from one place, a config value that gets swapped wholesale - those are fine.

But most state doesn’t stay that simple. You start with a single integer, it becomes a struct with three fields, and now you need to update two of them based on the third. At that point, Set(func(*T)) is the only safe option.

• The client disconnected
• A parent deadline expired
• The server started shutting down
• Some code somewhere called cancel() explicitly

Go 1.20 and 1.21 added cause-tracking functions to the context package that fix this, but there’s a subtlety with WithTimeoutCause that most examples skip.

What “context canceled” actually tells you

Here’s a function that processes an order by calling three services under a shared 5-second timeout:

func processOrder(ctx context.Context, orderID string) error {
    ctx, cancel := context.WithTimeout(ctx, 5*time.Second)  // (1)
    defer cancel()  // (2)

    if err := checkInventory(ctx, orderID); err != nil {
        return err  // (3)
    }
    if err := chargePayment(ctx, orderID); err != nil {
        return err
    }
    return shipOrder(ctx, orderID)
}

• (1) creates a derived context that automatically cancels after 5 seconds
• (2) cleans up the timer when the function returns, standard practice per the context package documentation
• (3) if anything goes wrong, including a context cancellation, the error is returned as-is

When a context gets canceled, the underlying reason is either context.Canceled or context.DeadlineExceeded. Libraries wrap these in their own types (*url.Error for net/http, gRPC status codes for grpc), but errors.Is still matches the sentinel.

So if checkInventory makes an HTTP call and the client disconnects while it’s in flight, the error that bubbles all the way up is:

context canceled

If the 5-second timeout fires while chargePayment is waiting on a slow payment gateway:

context deadline exceeded

Two sentinel errors. No reason, no origin, nothing. The caller of processOrder has no idea what actually happened.

You’d think wrapping the error helps:

if err := checkInventory(ctx, orderID); err != nil {
    return fmt.Errorf("checking inventory for %s: %w", orderID, err)
}

Now the log says:

checking inventory for ord-123: context canceled

Better. You know it happened during the inventory check. But you still don’t know why the context was canceled. Was it the 5-second timeout? A parent context’s deadline? The client hanging up? A graceful shutdown signal? The error doesn’t say.

Without the cause, you can’t tell whether to retry, alert, or ignore, and your logs don’t give on-call enough to triage.

When this happens in production, you end up scanning logs for other errors around the same timestamp, hoping something nearby gives you a clue. If the logs don’t help, you trace the context from where it was created, through every function that receives it, looking for cancel calls and timeouts. In a small service this takes a few minutes. In a larger codebase with middleware, interceptors, and nested timeouts, it can take a lot longer.

This has been a known pain point in the Go community for years. Bryan C. Mills noted this in issue #26356 back in 2018:

I’ve seen this sort of issue crop up several times now. I wonder if context.Context should record a bit of caller information… Then we could add a debugging hook to interrogate why a particular context.Context was cancelled.

– bcmills on #26356

On proposal #51365, which eventually led to the cause APIs, bullgare described the production experience:

I had a case when on production I got random “context canceled” log messages. And in the case like that you don’t even know where to dig and how to investigate it further. Or how to reproduce it on a local machine.

– bullgare on #51365

That proposal led to the cause APIs that shipped in go 1.20.

Attaching a cause with WithCancelCause

context.WithCancelCause gives you a CancelCauseFunc that takes an error instead of a plain CancelFunc. Here’s the same processOrder rewritten to use it:

func processOrder(ctx context.Context, orderID string) error {
    ctx, cancel := context.WithCancelCause(ctx)
    defer cancel(nil)  // (1)

    if err := checkInventory(ctx, orderID); err != nil {
        cancel(fmt.Errorf(
            "order %s: inventory check failed: %w", orderID, err,
        ))  // (2)
        return err
    }
    if err := chargePayment(ctx, orderID); err != nil {
        cancel(fmt.Errorf(
            "order %s: payment failed: %w", orderID, err,
        ))
        return err
    }
    return shipOrder(ctx, orderID)
}

• (1) cancel(nil) as the default, sets the cause to context.Canceled
• (2) before returning the error, records a specific reason that includes the original error via %w

Now you can read the cause with context.Cause(ctx). If checkInventory fails because of a connection error, the cause comes back as: