I’ve just released Kontiki, a Python microservices framework built on AMQP and asyncio.It is inspired by Nameko’s design principles, which I found very elegant, and aims to provide a coherent, production-usable developer experience around the same message-driven approach.

Sharing it here in case it is useful to others.

I start to learn Rust on my freetime since 2 years. And one of my pet project is a rust lib compliant with certain feature of Nameko, really inspire from rusty-celery. This project is Girolle. Girolle is far from been production proof but i already got win cases at work with it.
Thanks to all the contrubutors of Nameko.

Cheers.

I know it’s rather old topic, but out of curiosity have you finally setup a websocket client as dependency provider?

@app.get("/test")
async def proxy(request: Request):
    try:
        os_headers = []
       
        HTTP_SERVER = AsyncClient(base_url="http://127.0.0.1:5000/login",timeout=1000.0)
        url = httpx.URL(path='', query=request.url.query.encode("utf-8"))
        rp_req = HTTP_SERVER.build_request(
            request.method, url, headers=os_headers, content=await request.body()
        )
        rp_resp = await HTTP_SERVER.send(rp_req, stream=True)
        return StreamingResponse(
            rp_resp.aiter_raw(),
            status_code=rp_resp.status_code,
            headers=rp_resp.headers,
            background=BackgroundTask(rp_resp.aclose),
        )
    except Exception as e :
        print( e)

How can I do the same thing in Nameko?

Since in my case, I used “nameko run” to up a service class and what I observed that only one method was in use at a time, after I called many rpc at the same time. It seems to work like a normal queue. My attempt to update the max_workers parameter appears to have had no effect.

Thank !

Nameko can run multiple services inside the same runner without any problem. It’s often helpful to do this, for example when testing multiple services interoperating, but in production I recommend using a runner per service. That makes it easier to reason about things, and you generally get more capacity because (on a multi-core host) each service runner will potentially occupy its own core.

And I found that, it because I run rpc and event handler service in ServiceRunner. It turn woking well asynchronously when I run service Individually. I got the same behavior when I tested with the nameko shell to run services. So mostly we don’t run rpc and event handler service in the same ServiceRunner?

How are you running these services? It looks to me like you have not applied the eventlet monkey patch.

My requerment is just want to dispatch the event handler without waiting, but in my case after dispatching the event_handler the EventDispatcher is keep waiting until the event_handler done process. Is ACK will respond back after the process done? or is there any configuration that I missed ?

I have attached some references to my case, as you can see there is a sleep for 20s in test event_handler. When I dispatched this event, I expected it to take less than 20 seconds, but it almost never did. As I noted, the dispatcher will complete after the event_handler done its operation.

Event dispatching service

Event listening service

Reuslt

Addition
I expected this event dispatcher to work, such as when I wrap event dispatcher in thread, since it worked when I use eventlet.spawn(event dispatcher).

The event_handler entrypoint is async by default – there is no client waiting for an answer. The only thing that waits for the method to finish is the ACK of the event message. It is implemented this way to make sure that, if the worker is interrupted somehow, the message is requeued for another handler to pick up.

You can run the following in two identical processes and they’ll both happily listen:

import eventlet
eventlet.monkey_patch()

listen_sock = eventlet.listen(("127.0.0.1", 8001))

print("listening")

try:
    while True:
        sock, addr = listen_sock.accept()
        print("new sock", sock, addr)
except KeyboardInterrupt:
    print("quitting")

listen_sock.close()

On my MacOS machine, the last-started process will receive all of the connections; if you terminate that process, the other starts receiving them.

eventlet.listen() accepts kwargs reuse_addr and reuse_port that set SO_REUSEADDR and SO_REUSEPORT respectively on the listening socket. But because they’re both set, regardless of how you invoke the function, SO_REUSEPORT clobbers the role of SO_REUSEADDR in controlling whether or not there’s an error about the port already being in use. The upshot is that passing reuse_addr=False into eventlet.listen doesn’t actually prevent two processes binding to the same port, but passing reuse_port=False does.

The following Stack Overflow post has a really helpful breakdown of what these two options do: linux - How do SO_REUSEADDR and SO_REUSEPORT differ? - Stack Overflow

Clearly that should not be possible, but have also noticed this behaviour, and I’m confused by it too. I have assumed it’s some problem with eventlet’s .listen(), but I’ve not spent the time to reproduce it or track it down to specific versions of eventlet/Python.

1. You’re using Nameko’s high-level API rather than using pseudo-internals (.kill() on the managed thread is an eventlet API)
2. If you have multiple service instances, events will allow them all to participate, whereas threads are constrained to just the instance that spawns them.

The easiest way would probably be to communicate to the thread that it should terminate. The nameko-grpc library does something like this for the threads it uses to manage each connection. See nameko-grpc/connection.py at master · nameko/nameko-grpc · GitHub

It shouldn’t matter if config.get(‘SERVICE_NAME’) is None temporarily, as long as you reimport the module after the patch is in place.

What is the error you’re seeing?

Another question related to MyService above: What is the best practice to set the name of a service from a config object?

When service gets imported at import service # module where MyService is defined line, config object is None (config object is not patched yet here), and the line name = config.get(‘SERVICE_NAME’) runs into an error.

You can update the global config object directly from your test. In fact, the config object has a patch method for this purpose. I’ve written some docs on this here Testing services -.

Your service is reading from the config object at import time, so you need to make sure your test populates the desired configuration beforehand. Moving the import inline is sufficient, or you can use importlib.reload.

This works:

from nameko.testing.services import worker_factory
from nameko import config

import service  # module where MyService is defined

import pytest
import importlib

class TestMyService:

    @pytest.fixture
    def set_config(self):
        with config.patch({"SERVICE_NAME": "MyService"}):
            yield

    def test_sum(self, set_config):
        importlib.reload(service)
        svc = worker_factory(service.MyService)

        assert( svc.name == 'MyService' )
        assert( svc.sum(2,3) == 5 )

I am using v3.0.0rc11. My service is defined as below:

import logging

from nameko import config
from nameko.rpc import rpc

class MyService:
    name = config.get('SERVICE_NAME')
    logger = logging.getLogger(__name__)

    @rpc
    def sum(self, num1, num2):
        self.logger.info(f'Num1: {num1}, Num2: {num2}')
        return num1 + num2

I have written my unit test thus:

from nameko.testing.services import worker_factory

from MySvc.myservice import MyService

class TestMyService:
    def setup_method(self):
        self.cfg = {
            'SERVICE_NAME' : 'MyService'
        }

    def test_sum(self, container_factory):
        cfg = {
            'SERVICE_NAME' : 'MyService'
        }
        
        svc = worker_factory(MyService, config=cfg)

        assert( svc.name == 'MyService' )
        assert( svc.sum(2,3) == 5 )

The unit test fails, with this error:

nameko.exceptions.ExtensionNotFound: DependencyProvider(s) 'dict_keys(['config'])' not found on <class 'MySvc.myservice.MyService'>.

What am I doing wrong? How can I pass in config to worker_factory so the config gets passed in to service properly?

Thanks for all the help!

Radha.

So far, I’ve been able to spawn managed threads, but I’m looking for a way to stop them.

def start(self):
if not self._starting:
self._starting = True

So I understand that this code creates a listening socket only once allowing many @http entrypoints. So I understand how this works in one memory space of one process.

But how a completely separate process can also share WebServer’s SharedExtension!?

Thx

class Rpc(NamekoRpc):

def __init__(self, only_once: bool = False, *args, **kwargs):
    self.only_once = only_once
    super(Rpc, self).__init__(
        *args, **kwargs
    )

def handle_message(self, body, message):
    if self.only_once is True:
        # Consume only once, automatically ack after getting the message
        self.rpc_consumer.queue_consumer.ack_message(message)
    try:
        args = body['args']
        kwargs = body['kwargs']
    except KeyError:
        raise MalformedRequest('Message missing `args` or `kwargs`')

    self.check_signature(args, kwargs)

    context_data = self.unpack_message_headers(message)

    handle_result = partial(self.handle_result, message)
    try:
        self.container.spawn_worker(self, args, kwargs,
                                    context_data=context_data,
                                    handle_result=handle_result)
    except ContainerBeingKilled:
        self.rpc_consumer.requeue_message(message)

def handle_result(self, message, worker_ctx, result, exc_info):
    amqp_uri = self.container.config[AMQP_URI_CONFIG_KEY]
    serializer = self.container.config.get(
        SERIALIZER_CONFIG_KEY, DEFAULT_SERIALIZER
    )
    exchange = get_rpc_exchange(self.container.config)
    ssl = self.container.config.get(AMQP_SSL_CONFIG_KEY)

    responder = Responder(amqp_uri, exchange, serializer, message, ssl=ssl)
    result, exc_info = responder.send_response(result, exc_info)

    if self.only_once is False:
        self.rpc_consumer.queue_consumer.ack_message(message)
    return result, exc_info

rpc = Rpc.decorator

class GreetingService:
    name = 'greeting'
    @rpc
    def hello(self):
        return f"{datetime.today()}: hi"

Within the shell, I can call hello in a loop without problems. If I wait 5 minutes or more to call hello then the shell hangs indefinitely.

helo=n.rpc.greeting.hello
>>> helo()
'2022-10-03 12:40:12.628285: Hello, alarm device count=936'
>>> helo()
***hangs here***

The second helo will be stuck and impervious to keyboard interrupt. I have to kill the shell process.
Do you have any ideas why this is occurring?
Or do you have any suggestions for troubleshooting?

Environment:
nameko - 2.14.1
Windows 10
python3.9.5

from kombu import Queue
from nameko.messaging import consume

class Service:
    name = "sample_service"
    QUEUE = Queue("queue_name", no_declare=True)

    @consume(QUEUE)
    def process_message(self, payload):
        # Some long running code ...
        return result

By default behaviour, ACK will be sent to rabbitMQ broker after process_message function returns a response (Here, statement return result). I want to send an ACK as soon as consumer consumes the message. How can I do that?

*In library “pika”, consumer acknowledges as soon as message is consumed. That will be good example what I want to replicate with nameko’s consumer.

Thanks @mattbennett

nameko.config.patch({"AMQP_URI": "memory://"})

to test events without having a real MQ instance.

the issue is that the delay time is not reset after a successful call. With this behaviour you could basically create an indefinite sleep period if you don’t use the max_delay property, already after a few calls if you have a reasonable sized backoff time.

as the RetryDelay class is not locally scoped the state will preserved basically within your worker class. This is not a major issue however, if you have a few failures your sleep time will increase very rapidly for future failed requests.

I believe we should reset the delay to the initial value after a successful call. I don’t think the max delay is fair in this one as it will then basically replace the whole idea behind incremental increase of the sleeping time.

I can make a PR but I would first better understand why this has been designed in this way. At least it should be documented I think.

I’m trying to use Dask within a Nameko service, but I’m running into some issues.
Here is a sample code I use:

from nameko.rpc import rpc

from dask.distributed import Client
import dask.array as da

# creates a local Dask Cluster
client = Client()

class TestService(object):

    @rpc
    def task(self) -> str:

        dim = 100000
        chunk = 10000

        x = da.random.random((dim, dim), chunks=(chunk, chunk))
        y = da.exp(x).sum()

        y.compute()

Unfortunately, the following exception is raised at startup:

Traceback (most recent call last):
  
   ....

    main(args)
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/nameko/cli/run.py", line 179, in main
    import_service(path)
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/nameko/cli/run.py", line 44, in import_service
    __import__(module_name)
  File "/home/user/test_service/start.py", line 5, in <module>
    from test_service import TestService
  File "/home/user/test_service/test_service.py", line 16, in <module>
    from dask.distributed import Client
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/dask/distributed.py", line 11, in <module>
    from distributed import *
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/__init__.py", line 7, in <module>
    from .actor import Actor, ActorFuture
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/actor.py", line 5, in <module>
    from .client import Future
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/client.py", line 59, in <module>
    from .batched import BatchedSend
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/batched.py", line 10, in <module>
    from .core import CommClosedError
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/core.py", line 28, in <module>
    from .comm import (
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/comm/__init__.py", line 25, in <module>
    _register_transports()
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/comm/__init__.py", line 17, in _register_transports
    from . import inproc, tcp, ws
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/comm/tcp.py", line 387, in <module>
    class BaseTCPConnector(Connector, RequireEncryptionMixin):
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/distributed/comm/tcp.py", line 389, in BaseTCPConnector
    _resolver = netutil.ExecutorResolver(close_executor=False, executor=_executor)
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/tornado/util.py", line 288, in __new__
    instance.initialize(*args, **init_kwargs)
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/tornado/netutil.py", line 427, in initialize
    self.io_loop = IOLoop.current()
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/tornado/ioloop.py", line 263, in current
    loop = asyncio.get_event_loop()
  File "/home/user/.pyenv/versions/3.9.9/lib/python3.9/asyncio/events.py", line 642, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'MainThread'.

I worked this around by setting an asyncio event loop before dask.distributed is imported:

import asyncio

loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)

from nameko.rpc import rpc
from dask.distributed import Client
import dask.array as da
client = Client()

class TestService(object):
    ...

Now, when the RPC is called and the task is executed, I’m getting a different error:

ERROR:    | 2022-04-04 09:58:56.758 | nameko.containers::/...te-packages/nameko/containers.py:399 
error handling worker <WorkerContext [test_service.task] at 0x7fb47a9bb8b0>: 'coroutine' object is not iterable
Traceback (most recent call last):
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/nameko/containers.py", line 392, in _run_worker
    result = method(*worker_ctx.args, **worker_ctx.kwargs)
  File "/home/user/test_service/test_service.py", line 48, in pretend_task
    y.compute()
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/dask/base.py", line 288, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/user/test_service/.venv/lib/python3.9/site-packages/dask/base.py", line 572, in compute
    return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
TypeError: 'coroutine' object is not iterable
/home/user/test_service/.venv/lib/python3.9/site-packages/nameko/containers.py:417: RuntimeWarning: coroutine 'Client._gather' was never awaited
  del exc_info

I tried to place the Client() call within the RPC (among other attempts) but I clearly need to understand why I’m getting this error when interacting with Dask. Any suggestion? Thanks!

Is there a reason the project was discontinued? Seems like a neat tool to transition from django project to nameko microservices!

It seems that your autocomplete feature of you IDE is pointing you to another type of Config.

I’ve created a small test service with nameko 2.14.0:
service.py

from nameko.dependency_providers import Config
from nameko.web.handlers import http


class Service:

    name = "test_config"

    config = Config()

    @http("GET", "/")
    def foo(self, request):
        return f'Config Mongo url is {self.config.get("MONGODB_URI", "http://localhost:27017")}'

A config.yml

MONGODB_URI: "http://mongodb.com:27017"

And works as expected:

$ nameko run service --config config.yml

Try to run your service, it looks good. I don’t use pyright, but it seems to interfere with nameko’s config type on autocomplete.

Cheers!

I found this interesting question regarding nameko, sqlalchemy and psycopg2, but unfortunately, is unanswered: python - Potential Nameko concurrency issues with pyscopg2 driver for Postgresql - Stack Overflow

A brief excerpt of the question:

We are building a Python microservices application with Posgresql as service datastore. At first glance Nameko seems a good starting point. However the Nameko documentation section on Concurrency includes this statement […]
Our architect is suggesting Nameko is therefore not going to fly - because although the pyscopg2 Postgresql driver is advertised as thread safe:
With a link clarifying:

Warning Psycopg connections are not green thread safe and can’t be used concurrently by different green threads.

Personally, I have used Nameko-SQLAlchemy with Psycopg2 without problems, but I’m by no means a threading expert to know it’s safe.

What do you guys think?

Cheers!

Is it possible to know that a call comes via RpcProxy (inter-service) vs. ClusterRpcProxy (external service)? I know they both use the worker’s context_data, but I was not able to find an attribute that would help me to know from where the service call is coming.

For Integration Testing, container_factory can spawn multiple services and communicate within them, that’s why it needs a RabbitMQ connection.

Is there a reason you need container_factory for Unit Testing? You could spawn more than one instance with worker_factory, although without communication within them.

I am writing pytest based unit tests for my nameko services as outlined here: Testing Services — nameko 2.12.0 documentation. I’ve been using container_factory for testing my services for the most part. A few instances where I’ve used worker_factory.

However, it looks like there is no way I can execute these tests without a real rabbitmq instance. This is forcing us to tweak our CI infrastructure to accommodate this.

Is there a way to execute nameko unit tests without a real rabbitmq instance?

Thanks,
Radha

Since I have already switched to v3 (rc10), there only few things to be careful about.

• now you can use from nameko import config. Initially you were accessing it through container.
• Be sure to define variable AUTO_DELETE_EVENT_EXCHANGES at you config.yml file. Support for disabling event exchange declarations by mattbennett · Pull Request #730 · nameko/nameko · GitHub has been valuable to me.

That’s all at least for my case. I haven’t found anything else required to change after switching to 3.0.0rc10.

Happy upgrading

Spyros

You don’t hear from me often, because I rarely have any issues - which is a testament to the software you’ve written. I’m so glad I found it all those years ago…!

Geoff

Always good to hear your feedback

Yes, v3 is the way forwards. It is inching closer to being officially released, and certainly many users I know about now are using v3. There is now an upgrade guide too! You can read that on the draft version of the new docs https://agitated-hoover-1d4d78.netlify.app.

Matt.

Still loving Nameko. I’ve put off the move from 2 to 3 until release - but like BOTW2, it’s in limbo.

I’m starting a full refactor of the services I run that use 2, so I figure now is the best time to try out 3.

Are there any major changes to keep an eye out for? Aside from RpcProxy now being named ServiceRpc?

Hope all is well. 2020-2021 were strange years.

Geoff

i have same issue with training a ML model.

After train ended (about 1h) i received the error

Connection to broker lost, trying to re-establish connection...
Traceback (most recent call last):
  File "/Users/.../lib/python3.8/site-packages/kombu/mixins.py", line 193, in consume
    conn.drain_events(timeout=safety_interval)
  File "/Users/.../lib/python3.8/site-packages/kombu/connection.py", line 317, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/Users/.../lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 169, in drain_events
    return connection.drain_events(**kwargs)
  File "/Users/.../lib/python3.8/site-packages/amqp/connection.py", line 522, in drain_events
    while not self.blocking_read(timeout):
  File "/Users/.../lib/python3.8/site-packages/amqp/connection.py", line 527, in blocking_read
    frame = self.transport.read_frame()
  File "/Users/.../lib/python3.8/site-packages/amqp/transport.py", line 310, in read_frame
    frame_header = read(7, True)
  File "/Users/.../lib/python3.8/site-packages/amqp/transport.py", line 639, in _read
    s = recv(n - len(rbuf))
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 370, in recv
    return self._recv_loop(self.fd.recv, b'', bufsize, flags)
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop
    self._read_trampoline()
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline
    self._trampoline(
  File "/Users/.../lib/python3.8/site-packages/eventlet/greenio/base.py", line 211, in _trampoline
    return trampoline(fd, read=read, write=write, timeout=timeout,
  File "/Users/.../lib/python3.8/site-packages/eventlet/hubs/__init__.py", line 159, in trampoline
    return hub.switch()
  File "/Users/t.../lib/python3.8/site-packages/eventlet/hubs/hub.py", line 313, in switch
    return self.greenlet.switch()
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/.../lib/python3.8/site-packages/kombu/mixins.py", line 171, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/Users/.../lib/python3.8/site-packages/kombu/mixins.py", line 195, in consume
    conn.heartbeat_check()
  File "/Users/.../lib/python3.8/site-packages/kombu/connection.py", line 306, in heartbeat_check
    return self.transport.heartbeat_check(self.connection, rate=rate)
  File "/Users/.../lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 220, in heartbeat_check
    return connection.heartbeat_tick(rate=rate)
  File "/Users/.../lib/python3.8/site-packages/amqp/connection.py", line 767, in heartbeat_tick
    raise ConnectionForced('Too many heartbeats missed')
amqp.exceptions.ConnectionForced: Too many heartbeats missed

Can you tell me how you fixed it?

I am looking for best practices in implementation schemes for authenticating websocket communication. At the moment, this is the workflow I am considering:

1. Client (Web UI) gets a JWT from invoking a REST API endpoint - /login.
2. Client initiates a websocket session - subscribing to a set of events, passing the JWT (somehow).
3. Upon authenticating JWT, server begins sending events based on the socket connection.

What are the best practices for all passing JWT while initiating a websocket session? And how can I parse it at the websocket server end?

Appreciate all guidance in this matter.

Thanks,
Radha.