Abstract
This study introduces Conversation Routines (CR), a structured prompt engineering framework for developing task-oriented dialog systems using Large Language Models (LLMs). While LLMs demonstrate remarkable natural language understanding capabilities, engineering them to reliably execute complex business workflows remains challenging.

The proposed CR framework enables the development of Conversation Agentic Systems (CAS) through natural language specifications, embedding task-oriented logic within LLM prompts. This approach provides a systematic methodology for designing and implementing complex conversational workflows while maintaining behavioral consistency. We demonstrate the framework’s effectiveness through two proof-of-concept implementations: a Train Ticket Booking System and an Interactive Troubleshooting Copilot. These case studies validate CR’s capability to encode sophisticated behavioral patterns and decision logic while preserving natural conversational flexibility.

Results show that CR enables domain experts to design conversational workflows in natural language while leveraging custom functions (tools) developed by software engineers, creating an efficient division of responsibilities where developers focus on core API implementation and domain experts handle conversation design.

While the framework shows promise in accessibility and adaptability, we identify key challenges including computational overhead, non-deterministic behavior, and domain-specific logic optimization. Future research directions include CR evaluation methods based on prompt engineering framework driven by goal-oriented grading criteria, improving scalability for complex multi-agent interactions, enhancing system robustness addressing the identified limitations across diverse business applications.

UPDATE (18/02/2025)

The article has been published on arXiv! Compared to the original version, the updated version submitted to arXiv includes som experimental results, refinements and formatting adjustments: https://arxiv.org/abs/2501.11613

Conversation Routines: A Prompt Engineering Framework for Task-Oriented Dialog Systems was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

A few days ago, the just released SWARM open-source project [1] from OpenAI sparked quite a bit of discussion within the agent-based Generative AI community, particularly among those focused on conversational AI. It’s a small, simple project (so far) that the company defines as:

An educational framework for exploring ergonomic, lightweight multi-agent orchestration. It is managed by the OpenAI Solutions team.

When comparing SWARM to well-known multi-agent frameworks such as LangGraph, CrewAI, AutoGen, and others, many argue that there is nothing groundbreaking about this small framework, which appears more like a demo than a production-ready platform. Indeed, OpenAI itself tends to characterize its project as a simple cookbook [2].

In a certain sense, I strongly agree; however, several key concepts in SWARM’s proof-of-concept align with my perspective on constructing LLM-based conversational agents. It is important to clarify that the term (conversational) agent, as I use it, has a very specific meaning, which only partially overlaps with the concept of AI agents as understood in the LLM-based community recently. For a more detailed discussion, please refer to my previous article: Conversational Agent with a Single Prompt [3].

Agents = Routines + Handoffs

The accompanying insightful OpenAI github cookbook [4] highlights several key points. The framework introduces the concept of routines that embed conversational workflow logic in a no-code manner (as I previously referred to in my article, referring to this as Directive Instructions on Conducting the Dialog).

The fundamental premise of SWARM is to decompose a complex conversational workflow (macro-task) into multiple smaller tasks managed by agents, which can be viewed as LLM-based experts in specific domains and policies. These agents collaborate through straightforward yet effective handoff mechanisms based on function-calling design patterns. So far, nothing new — I agree [5][6].

Instructions (on Conducting the Dialog)

With SWARM, it is possible to define complex workflows where conversational designers (prompt engineers) articulate the business logic of the workflow in natural language. The related backend business logic components, referred to as tools within the context of LLM programming, remain separate and can reside in custom Python code (or any programming language of choice).

This allows applications — whether conversational or otherwise — to be constructed from distinct components: LLM-based workflows, developed by prompt engineers (or coders), and hard-coded programs, handled by traditional software developers.

To me, this is the most crucial aspect when building an AI team in practice: bringing together developers (sw coders) and conversational designers (~no-coders) to work collaboratively!

Let us now examine a simple explanatory code snippet extracted from the blog example:

Notably, in the example provided in the blog, the LLM-based routine may include fixed (deterministic) steps, such as mandatory dichotomous questions (yes/no), implemented as Python functions. In the snippet here above, two routines (agents), are defined: triage_agent and sales_agent, each possessing its own workflow as specified in the instructions prompt, along with a set of associated functions, commonly known as tools, which implement the relevant business logic on the backend.

No-code Instructions

The instructions consist of simple sequences of conversational step directives, written in natural language or pseudo-code (such as bullet points or any structure expressible in natural language), which may include conditionals and/or loops.

This is significant as it demonstrates a method for conceptualizing chatbot interactions that are not reliant on hard-coded scripts (in a specific programming language or dedicated conversation workflow tool) but instead are based on high-level directives for conducting dialogue, articulated in natural language within system prompts.

Deterministic Workflow Checkpoints

In revisiting the snippet analysis, the most significant feature is the execute_order() function outlined earlier within the sales_agent tool. The notable aspect of this function is that when the sales agent determines an order should be executed, it invokes the execute_order() function, which can prompt a yes/no confirmation request from the user.

SWARM thus enables a synthesis of no-code programming (implemented as instructions in prompts) with workflows that include hard-coded dialog turns (implemented as programming code in invoked functions). I refer to these functions as workflow checkpoints.

This approach is particularly noteworthy as it allows for the design of complex conversational applications where hard-coded workflows are seamlessly integrated with LLM-based workflows.

Context Variables and Task-Oriented State Machines

The framework introduces context variables — a simple yet effective mechanism for retrieving and storing contextual data shared across routines. While the implementation may appear basic, its straightforwardness is part of its strength.

Interestingly, opinions differ on how SWARM handles state management: some consider context variables as part of a potential state-machine-based approach, while others argue that the framework remains fundamentally stateless, given its dependence on stateless calls to the LLM models driving the agents.

From my perspective, particularly in conversational design, SWARM effectively implements a task-oriented state machine, albeit at a high level of abstraction. In this framework, conversational states are naturally encoded within the logic of routines, enabling both input and output data to be stored in shared context variables. This allows conversation designers to focus on agent-driven tasks without the need to explicitly conceptualize a full state network, as in the Langraph approach.

Additionally, the framework adopts a minimal yet effective testing method through evals. Once again, I appreciate this clear and practical methodology for validating routine behaviors.

Conclusion

While SWARM may not yet rival more established multi-agent platforms like LangGraph, CrewAI, and AutoGen (to mention the most prominent) in terms of sophistication, it introduces a promising approach to orchestrating LLM-based conversational agents. Its ability to decompose workflows into smaller, specialized tasks managed by individual agents demonstrates a practical framework for agent-based orchestration.

What may differentiate this approach from other frameworks is its emphasis on workflow development through the seamless integration of no-code and hard-coded processes. This allows conversational designers to define overarching logic using natural language prompts, while developers handle more complex backend functions through traditional coding methods.

This hybrid design creates a fluid workflow where the roles of both no-code and coded components are clearly delineated, allowing for flexible, collaborative development of conversational applications.

Moreover, SWARM’s minimalistic reliance on context variables and simple testing methodologies strikes a balance between simplicity and functionality, making it a pragmatic choice for developing medium-complexity agentic systems. Whether SWARM will evolve into a robust, production-ready framework remains to be seen. However, its approach represents an important step toward bridging the gap between conversational designers and software developers, fostering a more collaborative environment for building next-generation conversational AI systems.

As the community further explores its potential, SWARM may prove to be more than just an educational tool, but rather a viable option in the growing landscape of LLM-based multi-agent frameworks.

Update (2024–11–03): Insights from Initial Experiments

Following several hands-on tests with SWARM, I would like to provide further insights into my experience with this framework. The key advantage of adopting an agentic approach (exemplified by SWARM, along with other LLM-based agentic frameworks) lies in a principle reminiscent of the structured programming paradigm of the 1960s: break down the main problem (or workflow in this context) into smaller, manageable sub-problems.

Each sub-problem is assigned to a specialized agent, which operates based on instructions crafted by a conversation designer and utilizes tools or backend functions implemented in Python by a developer.

This approach represents a substantial advancement over the monolithic, LLM-based methods that were our primary focus just a year ago. For example, consider a Retrieval-Augmented Generation (RAG) application. In tests, I implemented a search workflow using cooperating agents aligned with SWARM’s design. In this setup, a master orchestrator agent delegates tasks to specialized agents, each tasked with retrieval from sources such as relational databases or vector stores. The results demonstrated minimal hallucinations and an effective response strategy for unmet requests. Overall, this multi-agent approach outperformed the single-LLM model.

Beyond the design pattern best practices mentioned above, I have reassessed my previously favorable view of what I termed “Deterministic Workflow Checkpoints.” Specifically, the example OpenAI Python function execute_order(), which directly reads user input (via input()) and writes output (using print()), illustrates a suboptimal design pattern. The hard-coded backend logic in this function bypasses both instructional flows and user interface design, leading to several limitations.

For example, in scenarios where user confirmation is required (such as verifying a purchase or payment), it is valid to include a checkpoint function that prompts a yes/no confirmation. However, it is not ideal for such functions to embed workflow interaction logic directly. Instead, it would be more effective to delegate interaction logic to a specialized user interface agent — this could be the main orchestrator agent or, even better, a dedicated agent attuned to specific user interface modalities (whether chat, voice interface, or email), the bot-persona design and so on.

While the principle of using functions as checkpoints is sound, the original OpenAI example relying on direct input() and print() calls could be considered an anti-pattern, as it does not conform to the separation of concerns in agent-based design.

Regarding LLM models, some have argued that SWARM is specific to OpenAI’s models, but this is not accurate. OpenAI API calls have become a de facto standard, and many non-OpenAI vendors provide compatible interfaces. In my tests, I used Azure OpenAI deployments, and the framework operated effectively with only few lines of additional initialization code I wrote, as shown below:

Lastly, you can utilize SWARM as-is on-premises (either locally on your hardware or in a private cloud) with the outstanding ollama engine, which supports lightweight open models, as demonstrated by Cole Medin in his YouTube video [7].

Last but not least, It is unfortunate that OpenAI appears reluctant to further develop the SWARM project on GitHub. Specifically, they have disabled the ability for users to open issues, which is an unusual practice for a prominent organization that incorporates the term “Open” in its branding.

Nonetheless, I remain appreciative of OpenAI’s contributions. I would like to express my gratitude for underscoring a principle in which I firmly believe: that LLMs serve as effective conversationalists and that the multi-agent approach supports the no-code aspirations of many chatbot conversation designers, enabling machines to facilitate dialogues on their behalf.

What do you think?

References

[1] OpenAI SWARM GitHub Repository: https://github.com/openai/swarm

[2] OpenAI SWARM GitHub Cookbook Blog Article: Orchestrating Agents: Routines and Handoffs https://cookbook.openai.com/examples/orchestrating_agents

[3] Concept of Conversational Agents: Conversational Agent with a Single Prompt? https://www.linkedin.com/pulse/conversational-agent-single-prompt-giorgio-robino-vrppf/

[4] OpenAI SWARM Cookbook Jupyter Notebook: https://github.com/openai/openai-cookbook/blob/main/examples/Orchestrating_agents.ipynb

[5] Microsoft Autogen Handoffs: https://microsoft.github.io/autogen/dev/user-guide/core-user-guide/design-patterns/handoffs.html

[6] Old OpenAI Prompt Engineering Tutorial: https://platform.openai.com/docs/guides/prompt-engineering/strategy-split-complex-tasks-into-simpler-subtasks

[7] Ollama + OpenAI’s Swarm — EASILY Run AI Agents Locally, by Cole medin:
https://www.youtube.com/watch?v=8jpVeUTNExI

SWARMing Conversational AI was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last weekend, just for personal fun, I conducted a non-scientific experiment to test how well the Large Language Models (LLMs) available on the market “know” a specific natural language (Italian, in my case) according to CEFR guidelines.

I’ll briefly recap what the CEFR classification standard is, detail the experiment, and finally share some thoughts about it.

What’s CEFR?

CEFR (Common European Framework of Reference for Languages), also known in Italy as QCER (Quadro Comune Europeo di Riferimento per le Lingue), is a standard for classifying language proficiency.

The CEFR is a guideline used to describe the achievements of language learners across Europe and increasingly worldwide. In November 2001, a European Union Council Resolution recommended using the CEFR to establish systems for validating language ability.

The six reference levels (A1, A2, B1, B2, C1, C2) are widely accepted as the European standard for grading an individual’s language proficiency. These levels cover several competencies: written and oral comprehension, and written and oral production.

• A1 and A2: Basic users who can understand and use simple phrases and sentences in familiar contexts. A1 indicates very basic comprehension and production, while A2 shows slightly more advanced skills.
• B1 and B2: Independent users who can handle more complex language. B1 users understand the main points of clear standard input on familiar matters and can produce simple connected text. B2 users comprehend and interact on a wider range of topics, producing detailed and coherent text.
• C1 and C2: Proficient users with advanced skills. C1 users understand a wide range of demanding texts, recognize implicit meaning, and express themselves fluently. C2 users easily understand virtually everything heard or read, summarizing information from various sources coherently.

The LLM Self-Evaluation Experiment

I collected the responses of each of the nearly 50 models available on the Chatbot Arena website to this question (prompt) in Italian:

“Qual è il tuo livello di conoscenza della lingua italiana, rispetto alla classificazione QCER? Rispondi solo con una parola indicante il tuo livello di competenza:”

Meaning in English: “What is your level of Italian language proficiency, according to the CEFR classification? Respond with only one word indicating your level of competence”.

For each model, I collected the response and had to elaborate a bit because, in some cases, the models replied with synonyms, long sentences, or nonsensical answers. I compiled all the results in a table. See the attached screenshots:

The table shows three columns: MODEL-NAME, QCER_LEVEL (the resulting CEFR level as a single word), and RANK (where 1.0 means the LLM replied exactly with the expected level; between 0.0 and 1.0 means the LLM replied with a synonym or a long sentence).

Almost half of the models, including small and open-parameter models, self-evaluate with good or very good proficiency.

Hmm… Maybe there is a bit of overestimation in these LLM self-assessments?!

Automating Language Proficiency Assessments?

My experiment was clearly just a curiosity! I’m not a linguist, so a more thorough investigation should be conducted by domain experts (linguist researchers).

The quick test I conducted was done via the chat (textual) web interface, so the models can’t estimate listening and speaking! I admit my simple prompt biased the LLMs to reply with a single word that we assumed valid just for reading/writing abilities. A complete evaluation would require interfacing these LLMs with a voice interface (to test listening and speaking) and the production of content with varying levels of difficulty.

With a more scientific evaluation, a human language expert, typically a language teacher from a certifying body, could examine an LLM to classify proficiency using the same CEFR metrics we use for humans (reading comprehension, writing production, listening comprehension, speaking production, interaction, mediation).

A further step could be to develop a comprehensive LLM-based testing application to fully automate the proficiency examination, using a top-level LLM as the examiner to test other LLMs (acting as examinees). So, perhaps, by using high-proficiency LLMs as CEFR experts, we could automate some of the real teachers’ work on CEFR examinations, evaluating the language proficiency of human students… or other LLMs.

More generally, many e-learning activities (language-learning related and beyond) usually done by human teachers could be partially implemented by LLM-based conversational agents.

For example, teachers could be assisted by conversational assistant applications to manage the “heavy work” of exercising students and examining them, reporting relevant milestones and events to the teacher. There is a vast range of automation in the edutech sector that is now enhanced by generative AI.

What do you think?

#GenerativeAI #LLMs #LargeLanguegeModels #LanguageLearning #eLearning #EduTech #CEFR #QCER

Testing the Language Proficiency of Popular LLMs was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

Using Large Language Models for Chatbot Development: Specializing in Prompt Design

In this article, I share my experience in constructing Generative AI prompts to develop Conversational Agents.

First, I will clarify the relevant terms. Then, I will provide a brief overview of how we can utilize Large Language Models (LLMs) as intelligent conversationalists. Finally, I will present some compelling use cases where I have refined prompt engineering best practices to implement chatbots solely from no-code requirement specifications (system prompts).

Conversational Agents

Years ago, during my previous academic career (specifically as an assistant researcher at ITD-CNR), my research leader and other researchers always referred to chatbots as conversational agents. This perplexed me, as I’m particular about terminology in computer science. I always understood an agent to be any kind of software that intermediates among humans to perform some task (usually delivered by a human).

My point was that not every chatbot is truly an agent in functional terms.

For example, consider a voice system that acts as an assistant (nowadays we might call it a voice copilot) for a worker, assisting them in accomplishing specific real-life working tasks. Is it correct to define this system as a conversational agent? Maybe not, because it lacks agentive functionality. The term assistant may be more appropriate for augmented-reality scenarios like this (read also my previous article, Voice-cobots in industry. A case study). However, I admit that historically, in the scientific and academic community, conversational agent and chatbot have been used as synonyms.

Nevertheless, things have become more confusing with recent advancements in LLM-based autonomous agents. In this research area, which is broader than conversational applications, agents can autonomously define and execute micro-tasks based on a human-provided description in natural language (the system prompt) of a specific high-level duty or activity. This is a fascinating area of research with potentially disruptive practical applications, and there are many software frameworks available, but that’s a slightly different topic. Let’s now focus instead on the conversational application verticals.

Overall, I use the term Conversational Agent to refer to a specific type of agent that performs conversational tasks on behalf of a human.

Progress with LLM-based conversational agents allows us to build chat systems with a single prompt based on cognitive architectures. By utilizing advanced state-of-the-art LLMs, developers can describe what the chatbot should do without having to program the conversation as a series of fixed dialog states. From the development perspective, this could be a definitive cost-saving alternative to solutions based on intents, slots, states, and hard-coded flow management.

LLMs as Core Layers for Agent Engines

Long story short, GPT-based Large Language Models have revolutionized the field of conversational AI since the release of GPT-3 by OpenAI. These recent LLMs, trained on vast amounts of text data, can generate human-like responses and engage in meaningful dialogues. Their ability to understand and generate language makes them ideal for building chatbots.

Instruction-based Chat Completion Models

A basic foundation model (a large language model trained with sufficient data to ‘know’ a specific human language) is not sufficient to be a valid engine, capable of making conversations and reasoning.

Simply put, the disruptive improvement in GPT-3 models occurred with GPT-3.5-turbo (the model behind the famous ChatGPT, see my previous article: Reflecting on ChatGPT’s Anniversary). GPT-3.5-turbo is based on the foundation of GPT-3 but enhanced by a supervised training algorithm (HFRL and similar supervised training mechanisms) that enables it to converse with people in a fluid natural language, using polite and ‘controlled’ manner.

More importantly, the models from GPT-3.5 onwards are also instruction-based models because they are trained with programming code (OpenAI coined the term instruct). This last feature enabled some sort of ‘reasoning’ abilities. The LLMs are now able to perform themself some programmatic ‘logic’ understanding, such as concepts of programming languages including sequences, conditionals, and iterations.

Function Calling Feature

Another disruptive feature that nearly all state-of-the-art generative models now possess is the ability to call external functions/APIs (sometime called tools in LLM agents jargon). This is achieved through special fine-tuning of the aforementioned models, enabling LLMs to ‘call’ external functionalities, such as programs made in any programming language, to solve specific requests or actions and retrieving real-time data. This is a fundamental need in a cognitive architecture, where the LLM is the core ‘reasoning’ component that autonomously retrieves information from external systems or invokes actuators.

The function-calling feature is crucial for autonomous agents but not essential for building basic conversational agents. However, function-calling becomes a must-have when the conversational system needs to invoke external APIs. For example, a customer care assistant might need to open a ticket in an internal ticketing system or query the system to monitor the ticket status and inform the customer during the conversation.

The recent generative language models (instruction-based LLMs fine-tuned for chat completions, also enabled by function calling) can understand logic and instructions (through directive written in natural language in the prompt) and have an improved capacity to conduct human-like conversations in nearly any natural language. Additionally, these models can interact with external (proprietary) APIs. All in all, today’s models like GPT-4 or equivalents are viable engines for building autonomous agents capable of performing task-oriented conversations typically handled by humans.

In the next paragraphs, I will delve into this with some examples, but first, I will introduce the prompt engineering approach I used.

Prompt Design for Task-oriented Conversations

Prompt engineering is the practice of designing and refining input prompts to effectively guide the behavior and output of language models. By carefully crafting these prompts, users can enhance the model’s ability to understand and respond to complex instructions, ensuring more accurate and contextually appropriate outputs. This technique is crucial for optimizing the performance of state-of-the-art generative models, enabling them to perform specific tasks, generate creative content, and simulate ‘human-like’ conversations with precision.

The techniques I experimented with are about writing system prompts to instruct the LLM to conduct conversations in specific application domains to achieve particular tasks.

In-Context Learning

In all use cases I’ll introduce, I used a similar approach: the system prompt is composed of an introductory context section where I defined 1: the goal of the conversation (or task), 2: the bot-persona (the description of the agent’s characteristics/character, using the usual conversation design metrics), 3: the user persona (a description of the user profile), 4: The core part of the context is contextual data useful for the current conversation session. For example, if the conversation is an interview for a job applicant, this data includes the job description file and the candidate’s curriculum. More generally, the technique is akin to the one made famous with Retrieval Augmented Generation (RAG) applications, where you ‘stuff’ inside the prompt data retrieved (maybe with some embeddings database or any specific vertical data retrieval system).

When considering the data needed to accomplish a task-oriented conversation, it could be anything that fits into the prompt context window size (4K tokens, 8K tokens, 16K tokens, and so on). In all practical use cases I have experimented with and mentioned below, a context window of 4–7K tokens has been entirely sufficient for the purpose.

Directive Instructions on Conducting the Dialog

After the context part of the prompt, in the following instruction section, I detailed the required steps (actions to be accomplished in a specific order). This is the tricky part where you instruct the model not just on how to conduct the conversation in terms of social practices and human conventions, but also provide guidelines regarding the topics to cover, possibly including explicit questions or general behaviors to adopt.

Here, you instruct the LLM on what topics must be covered in the chat, how to conduct the dialogue with more or fewer guidelines, and how to guide the conversation from point A to the desired point B. Finally, the instructions must include criteria for deciding when to end the conversation session, which depends on the specific application and can be a bit tricky to implement.

Some Application Use cases

I introduce three dialogue systems I prototyped for entirely different verticals. All these applications have in common the fact that I wrote the conversation program as a single system prompt for an LLM. In chronological order of my developments:

Case 1: A Virtual Caregiver for Patient Telemedicine Visits

I have been involved in some prototypes related to the healthcare vertical, specifically in transcribing and extracting data from practitioner-patient visits for Conversational Analysis (CA) using LLMs. As a side project, I developed an emulation of a remote monitoring visit where a virtual assistant (acting as a practitioner or caregiver) contacts a patient every day via an instant messaging app to monitor their health status, particularly considering the patient is potentially affected by COVID-19. The virtual caregiver asks the patient about their health status, chats with them in a very natural way, delves into symptoms, and engages in small talk if the patient initiates it, while keeping the conversation focused on retrieving certain parameters: health status, temperature, blood oxygenation, and a few other variables.

Once all the requested information is retrieved, the virtual caregiver says goodbye to the patient and closes the conversation, internally returning a data structure (a JSON) containing all the information obtained from the patient. Interestingly, in this case, the end of the conversation is not strictly necessary. After the initial session, the user can re-engage with updates on their symptoms. The virtual caregiver replies to any patient questions or statements about their symptoms and internally emits any data updates via a function call. This example is also interesting for its psychological support aspects, but that’s another story.

You could argue that the described conversation is just an old-fashioned form-filling that one could implement with a simple hard-coded chatbot. However, the novelty of the LLM-based conversation is the naturalness of the interaction. This variance in how the system conducts any new conversation session, allowing user digressions while returning to the programmed goal of gathering information, is invaluable!

Case 2: A Customer Care Assistant

This is a classic chatbot application that I already mentioned in the article. Imagine a virtual assistant helping an employee of a very large company submit requests or report issues that can be tracked by opening tickets on a specific backend system. The user must also be able to ask about the status of previously submitted tickets. This is a very common chatbot application that I delivered in production as a standard state-machine flow tool seamlessly integrated with external REST APIs.

Subsequently, I tried to re-implement the same application using an LLM-based approach. The initial application involved highly constrained workflows, so what’s the advantage of using an LLM as a dialog conductor? I also struggled with implementing these programmatic steps that are simple to implement with a hard-coded flow. So, what are the advantages of implementing all this logic with a ‘declarative’ approach instead of using a standard software program?

There are two interesting pros: the conversation built by the LLM seems more ‘natural’, emulating the behavior of a human being (e.g. a help desk operator), allowing the user to describe an issue in various ways and guiding them to explain the problem concisely to gather all the necessary data.

The second advantage is the reduction in development time: with the single-prompt approach, the chatbot developer is no longer a software programmer using a chatbot development tool, but rather a prompt engineer with conversational design skills, who writes the chatbot specification as a special text in a natural language (English, Italian, etc.).

Besides the prompt engineer, we still need a backend developer who knows how to integrate external APIs, but what’s nice is that these two roles are quite distinct, and the software responsibility boundaries are clear.

Case 3: A Virtual Job Position Interviewer

The most fun and intriguing application I’m experimenting with is in the Human Resources vertical. Using the usual in-context learning prompt-writing approach, I built an emulator of a recruiter conducting an interview with a person who applied for a certain job position.

In the prompt context, I included the job post description and the candidate’s curriculum vitae. In the instruction section, I taught the LLM to act as a perfect recruiter, asking questions to verify all matches and mismatches between the role description and the candidate’s experience. The results are very impressive, and the virtual interviewer’s behavior is smart enough to detect weaknesses and strengths of the candidate by comparing the CV with the required skills. As a test candidate myself, I have been unable to lie in response to such precise investigative questions.

Since I’m not an expert recruiter myself, my approach could surely be improved with input from a domain expert in human resource recruiting. Nevertheless, my current experiments are astonishing. The system conducts a natural (similar to a human-to-human dialog) yet very rational interview, exploring points of weakness and verifying the truth of user statements in a polite and positive manner (as I instructed the bot-persona to do).

Besides the above application, I also created collateral LLM-based tools, such as a ‘pre-interview’ prompt to decide if a candidate deserves to be interviewed and some ‘post-interview’ tools that analyze the interview dialog and produce a structured report with a final ranking, but these are collateral (one-shot) LLM-based applications.

Prompt Development Challenges

LLMs are not deterministic. This has certain advantages, such as enabling smooth, fluent, always slightly different conversation variations, but it also presents some drawbacks. When considering the applications covered here, this randomness can potentially create issues. The main challenge I encountered was not the outcomes of the first prompt I designed, but the subsequent editing required to refine it to adjust some incorrect or unexpected runtime behavior.

Related to this, LLMs suffer from what I call fragility syndrome: you may have an initially well-functioning prompt, but even a minor, seemingly insignificant modification of a statement or a typo (by the way, typos are absolutely forbidden when writing prompts; please use a spell checker!) can cause different and unexpected runtime behaviors. Fixing this usually requires a lot of time spent on trial and error, where I rethink the prompt and often have to rewrite or reorganize it following a new, more logical approach.

For the prototypes I created, I admit I did not use any automated testing tools to validate the LLM outputs. This automatic evaluation is not a trivial task, although there are some emerging tools that can help prompt engineers validate prompts (this is a topic for a future article).

Tentative Conclusions

There is a lot of hype around optimal use cases for LLMs. Since 2023, I have seen hundreds of papers, articles and videos concerning RAG/LLM applications. While LLM-enabled data retrieval-based chatbots are certainly an important use case, for me, as a conversation designer also, the perfect use case for state-of-the-art LLMs is to exploit the conversational capabilities embedded in LLMs trained and fine-tuned on human conversations.

The no-code dream of developing chatbots has now become a reality with just prompt engineering skills?

What are your thoughts?

#promptEngineering #LLMs #generativeAI #GenAI #nocode #conversationalAgents #AutonomousAgents #chatbots #ConversationDesign #AI #MachineLearning #NaturalLanguageProcessing #AIChatbots #AIApplications

A Conversational Agent with a Single Prompt? was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

ChatGPT celebrates its first birthday on November 30, 2023. Exactly one year ago, OpenAI announced the birth of ChatGPT in a blog post, which you can find here.

In December 2022, the ChatGPT craze took off immediately, with hundreds of thousands of users and developers (like myself) eagerly engaging with this innovative technology.

I must confess that before ChatGPT was born, I was skeptical of large language models (LLMs) and statistic-based AI technologies in general. I followed the debates surrounding GPT-3 in 2022 with keen interest, and I must admit I was quite perplexed about the potential successful evolution of what some referred to as ‘statistic parrots’.

However, everything changed with the release of GPT-3.5 by OpenAI and subsequent models. These new models were extensively trained not only with human texts, including narratives and conversations, but also with programming code (InstructGPT). This has been a game-changer, in my opinion. These LLMs have been ‘filtered’ (with RLHF techniques, etc.) and fine-tuned for a chat experience interface (ChatGPT), not only demonstrating a nearly perfect natural language syntax but also exhibiting remarkable semantics, as acknowledged even by Walid Saba.

Inspired by amazing David Shapiro’s live coding videos, I delved into prompt engineering techniques in December 2022. Throughout 2023, I focused on exploring the applications of large language models, specifically in the realms of natural language processing and conversational interfaces, especially chatbots.

In my free time, I also studied a lot and participated a little bit in the growth of some excellent open-source Python development frameworks like LangChain, LlamaIndex, LiteLLM, just to name a few. I have been overwhelmed by the sheer number of academic papers published at an insane rate throughout the year.

Now, a crucial development is that the latest GPT-based models have demonstrated the capability to perform some reasoning-based tasks.

Many researchers are currently grappling with the challenge of constructing ‘autonomous’ agents, or more accurately, copilot assistants — systems designed to interact in real-time with users and collaboratively execute workflows alongside humans. This is a topic that has interested me for quite some time, as I developed in 2020 a research project concerning a voice cobot in logistics industrial real-time workflows.

I am currently intrigued by the prospect of utilizing LLMs as foundational layers in cognitive architecture frameworks to develop a new era of goal-oriented conversational agents. These systems engage in real-time conversations with users, perhaps in voice-first augmented-reality applications, incorporating voice, chat, vision, physical sensors, actuators, to achieve user goals without depending on hard-coded logic (an ‘imperative-driven’ software written in some programming language).

Perhaps we have reached a juncture where we can design these systems, setting ‘business requirements’ almost entirely in natural language, maybe by the final user directly (or probably helped by a prompts design expert). Are we witnessing the realization of the ‘no-code’ dream that some of us fantasized about just a few years ago?

Surely, 2024 will be an exciting year for the conversational AI community!

What are your thoughts?

Reflecting on ChatGPT’s Anniversary was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

English stands as the most widely utilized language on the internet, particularly in Western countries. It unquestionably serves as the lingua franca within the fields of computer science and scientific communities across the globe. The detailed breakdown of languages and their respective proportions in the training data for models like GPT-3.5 remains undisclosed by OpenAI and other providers.

Most contemporary large language models, including GPT-3.5, are likely trained on an extensive and diverse corpus of internet text, encompassing content from a multitude of languages. It is a rough assumption that the percentage of training data follows the language prominence on the internet. Please refer to the table below for approximate reference.

If we limit the languages to Western countries, we would probably see a higher percentage of English, possibly exceeding 50%. Do you concur?

What astonishes me, however, is the near flawless proficiency of state-of-the-art LLMs (Large Language Models) in minor languages, including my own: Italian. Rarely do I encounter syntax and “semantic understanding” errors, even in conversations conducted in highly proficient Italian.

This is undeniably remarkable!

So, when it comes to constructing complex LLM-based applications in a non-English language, one might assume that employing prompts exclusively in the non-English language is the straightforward and convenient approach. Nonetheless, I’m not entirely certain that it consistently produces the best results.

At first glance, there doesn’t appear to be a significant qualitative difference when comparing applications generated using prompts in Italian versus English.

However, let’s take a moment to consider some important factors in the equation: token usage, cost, latency, procedural understanding, language subtleties, and proficiency.

Tokens usage

Several months ago, I participated in a small Twitter thread where Hassan Hayat (@TheSeaMouse) shared a small experiment demonstrating that, when given the same text (an abstract of a famous paper in that case), GPT consumes fewer tokens when processing English compared to other languages:

Giorgio Robino on Twitter: "Yes! Italian even more: 465 tokensMaybe the translation test is not perfect but gives the idea. It seems to me that token composition is based on ASCII char encoding. E.g. accented vowels (é, à, etc) count 2 tokens and in general Italian words are longer than English pic.twitter.com/Fr7D7eIP2Y / Twitter"

Yes! Italian even more: 465 tokensMaybe the translation test is not perfect but gives the idea. It seems to me that token composition is based on ASCII char encoding. E.g. accented vowels (é, à, etc) count 2 tokens and in general Italian words are longer than English pic.twitter.com/Fr7D7eIP2Y

Well, token usage significantly increases for languages with non-Latin character sets. That’s partially expected, but what surprised me was that the original text, when translated into Italian and tokenized, required almost double the number of tokens compared to English.

In other words, a prompt written in Italian costs twice as many tokens as one written in English. Quite interesting!

Money Cost and Latency

These variables are closely tied to token length. If you’re using a cloud-based LLM with a pay-per-usage pricing model (say Azure Openai deployment), the more tokens you employ, the more you pay. However, even if you’re running an on-prem model at home, such as a LLAMA 70B or similar, the number of tokens you process represents an indirect cost.

More tokens processed translate to increased computation, which, in simplified terms, results in longer latency. This latter point is crucial, particularly when your application involves a conversational interactive system like a chatbot or, even more, a voice-interfaced assistant, where the latency is a crucial factor.

By the way, I don’t have any benchmarks or comparisons regarding the relationship between context window token size and latency. Please do let me know if you come across any relevant research or discussions on this topic.

Efficiency in ‘Procedural Understanding’

I’ve been experimenting with LLM prompts that implement conversational agent goal-oriented workflows, such as the common customer care use case where a chatbot guides the user to open a ticket, among other tasks.

In such scenarios, your goal is to instruct an LLM to follow a procedural workflow. This workflow may involve conditional statements, a sequence of actions such as slot filling, API requests, and even the generation of events in structured formats like JSON. For an academic example, refer to:

TOPIC: Opening a Support Ticket
STEP-BY-STEP WORFLOW:
1. Begin the process of opening a support ticket for the user's issue.
2. Initiate a conversation to gather all the necessary details.
3. Collect the following attributes one at a time:
   a. Issue Description: Ask the user to provide a detailed description of the problem they are encountering.
   b. Product or System: Inquire about the name or model of the product or system they are using.
   c. Contact Information:
      i. Choose a preferred method of contact:
         - If "email" is selected:
           - Request the user's email address.
           - Confirm the provided email address.
         - If "phone" is selected:
           - Request the user's phone number.
           - Confirm the provided phone number.
4. Display a summary of the gathered information and request confirmation from the user before proceeding.
5. Finally, submit the support ticket with the provided information. Generate the following JSON code without comments:
   {"api": "open_ticket", "email": email, "phone": phone, "description": description, "product": product}

It has come to my attention that these pseudo-code prompts are better understood when written in English. This may be because recent LLMs are also trained in programming languages, where the most commonly used programming terms are in English.

I’m not entirely certain about this observation, and I lack quantitative data for a definitive comparison. It’s more of a personal impression, and I would appreciate it if you could share any research studies on this topic.

Temporary Conclusions

My current approach is to write prompts in English, even for Italian language LLM-based applications, whether they are conversational systems or involve more complex tasks in the Italian natural language application verticals (such as meeting transcript summarizations or spoken dialogue conversational analysis).

By the way, to prompt the LLM to reply in Italian, I simply use a straightforward instruction like this:

LANGUAGE: Conduct the conversation in informal, fluent Italian.

This approach apparently offers several advantages:

• It minimizes token length, thereby reducing costs and latency.
• It maximizes procedural comprehension when it comes to workflow instruction prompts. Debatable.
• It allows for the creation of multilingual applications by design; you write your prompts once in English and can deploy your system in any language with a single word substitution!

However, I acknowledge that all of my considerations thus far have been quite qualitative and are based on my empirical experiments. I invite you to share your experiences and any relevant scientific evidence on the discussed topics.

Thank you for taking the time to read this article. Your feedback is highly valuable to me, so please feel free to leave a like and a comment below to share your thoughts and insights.

Giorgio

Non-English Languages Prompt Engineering Trade-offs was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

I started writing this article almost two years ago, and from time to time the topic comes in my mind, so maybe the question in the title is still valid.
In the first part of the article I analyze what is happening in the smartspeaker-based voice-enabled assistants market, and in the second part I introduce some change requests that could allow Telegram to become a winner competitor in this restless vertical.

Let‘s recap the current smartspeaker market landscape

For a few years, Amazon Alexa and Google Assistant have been competitors in the consumer smartspeaker market, with a similar market share. To be precise, Amazon Echo devices are a bit more spread all around the word, anyway the gap between the two competitor is not so huge.

Bixby has been a possible emergent third competitor, but Samsung seems have given up with the release of a branded smartspeaker device and the company has reserved the technology just for its branded smartphones.
End of the game of a splendid conversational technology conceived by Viv Labs’s Adam Cheyer and others.

After more than a year of stagnation, things are now changing because recently Google announced to sunset conversational actions. This means that:

In almost one year, Google Assistant will no more support actions designed only for smartspeakers.

Google focus on Android actions is announced and foreseen since a while. But now the company seems to have suspended his investments in voice technologies through smartspeakers.

Recently Sonos announced a new “service”, its own voice assistant that could be a competitor of Alexa (de facto, the current monopolist). But it’s not clear to me if the Sonos service will be implemented as an hardware device (an HI-FI smartspeaker / home multi-hub, aka the “soundbar”?).
The company, after having acquired in 2019 great snips.ai opensource-based company, will certainly focus on data privacy, proposing some local device processing of user voice data (on the smartspeaker, not on the cloud!).

Apple has his Siri voice assistant, running on their HomePod devices, right! But so far these Apple smartspeakers haven’t achieved an important market share. The first release of HomePods was too expensive and the Siri third party applications integration has never gained huge success.

Microsoft smartspeaker for Cortana was never born and the entire Cortana project seems to be died. Game never started.

Last but not list, also Facebook, pardon, Meta, is apparently at stake:

who remember Facebook Portal devices?

And what happened with the Facebook-Amazon agreement that established the coexistence of Facebook Portal and Amazon Alexa on the same devices produced and sold by Facebook? Hasn’t been heard from since! Also,

Whatsapp for business has not shined in the spaces of enterprise chatbot solutions.

Facebook, sorry Whatsapp, selected a short list of system integrator companies (2nd party) to “filter” 3rd party enterprise companies. In my opinion this has been an unsuccessful path that duplicated data privacy concerns and over-complicated accounting requests, creating a lot of frustration. By example, so far it’s impossible to set-up a chatbot on Whatsapp even for academic or no-profit purposes (e.g. my accounting access request to Whatsapp for CPIAbot, coming from myself as a public national research institute researcher, has never received a reply). Unclear business strategy.

Is the unique voice assistant a failed model?

So why the smartspeaker-based voice integration assistants are in this stagnation? In this second quarter of 2022, we are in a situation were

Amazon Alexa seems to be the only competitor left in the market of voice assistants based on smartspeakers.

But the question now is:

Has a unique smartspeaker-based central voice assistant a future (as a main hub at home or in the office)?

I remember discussions of experts few years ago, when pretty all agreed with the fact people need (at home) a unique voice assistant, not many assitants!
Is it still so?

Alexa now seems the winner of this prediction, the unique “1st party” assistant, the only one that leads the market. But this winning is maybe more related to the fact that Amazon competitors are “loosing out”. In fact also Alexa devices and investments didn’t grow as we expected. I have the feeling of a general stagnation also inside Alexa departments, by example the developers communities are no longer supported as they were years ago and many smart guys moved from Alexa to AWS. Weird, but significant signal.

So the winner-take-all desire of all above mentioned companies is probably failing and there are many reasons. All these big companies pretended to be the exclusive winner, proposing “walled gardens” and cloud-based ecosystems initially perceived by the market as disruptive, but these models are failing:

• Regarding final users, because the privacy data concerns haven’t yet solved (also because the cloud-based architecture proposed).
• Regarding 3rd party developers companies, because not proposing to them a complete technical advantage and a clear and profitable business model

My personal vision is that we do need a completely different approach, that exits from big player proprietary walled gardens, with two fundamental requirements:

• Hardware: we preferably do need a open-hardware smartspeaker device and some embedded open-software (and common protocols) on top.
• Software: we do need an open-architecture based on the coexistence of “peer-to-peer” multiple (voicefirst) assistants, operating on a common open architecture.

In other words,

• As a final user you wish a smartspeaker where you can use one or more voice assistants (made by “3rd parties”)
• As an application developer (service supplier/enterprise company), you wish a common protocol to plug-in your service into the above open-hardware smartspeaker ecosystem.

What does all this have to do with Telegram?

Even if the famous instant messaging (mobile) app is still a cloud-based closed-source system, an important and well engineered feature is the “by-design” possibility to enable 3rd party bots, in part following the multi-assistant architecture I mentioned above. Let’s deep dive!

What’s Telegram and what Telegram Bot APIs are?

Telegram is, for many reasons and above all user experience and proven security, probably the best instant messaging app available on common smartphones and personal computers operating systems!

This is not only because of the usual reasons Telegram fans mention when comparing this app with unloved Whatsapp, but, for developers, Telegram is great mainly because they can easily build chatbot applications, the so called Telegram Bots.

As all you programmers know, Telegram supplies a totally free, easy and performance-based way to build chatbots, using well done Bot APIs (just updated to version 6.1 in June 2022). You can set-up your chatbot in a few minutes using some high-level APIs on your preferred programming language. Last but not least, Telegram generously allows you to store for free a really huge amount of gigabyte of file storage. So far so good.

My experience as researcher and developer of CPIAbot

When I was researcher at ITD-CNR, from 2018 to 2020, I conceived and implemented CPIAbot, a Telegram voice-chatbot to help foreigners (emigrants) students of CPIA (Italian public adult schools) to learn Italian language as a second language at a basic level (L2/pre-A1).

As you can imagine, for a person that has to learn a foreign language, the verbal/spoken language understanding is a fundamental goal. So I allowed my chatbot to get inbound voice messages and to reply with outbound (synthetic) voice messages, back to the user. BTW, following research statistics we elaborated in the experimentation phase, the voice channel has been the preferred way for learners to interact with the bot!

From the UX perspective, using CPIAbot, students send text and/or voice messages to the bot. A custom server produces a text transcript with an ASR engine (at times I used pretty good Facebook wit.ai free service). Afterward the user transcript is elaborated by a dialog manager engine (my own opensourced naifjs) and the response is returned to the user as a text and/or voice message again, using a Google TTS voice, or a human-spoken audio recording. More info about CPIAbot on my old article here and on the academic project home page.

What’s missing in the almost apparent Telegram Bot API perfection?

So far you can develop a Telegram voicebot following the message-based paradigm. That’s not totally natural, I have to admit (from the natural language spoken interactions), even if nowadays people is used to communicate exchanging (also spoken) messages through instant messaging apps. Let’s consider audio messages ok for now.

Now I claim some change requests that could bring Telegram to be a competitor of big-players voice-assistants masterbots (someone says metabots) I mentioned above. Simply speaking:

Imagine your Telegram app as a smart-speaker embedded on your phone, able to connect you to any 3rd party bot without being an all-around “masterbot” assistant.

Sound good! Isn’t it? But how would it work in practice? To enable a great voice interface user experience, some features are now missing.

Change request #1: 🔊 Voice/Audio Messages Auto-Play

You need that when a bot is answering user (spoken) request, the bot voice message (response) is auto-played by the device (just when you are currently interacting with). Now, instead, user must click on the voice message icon to play it.

Of course, also for trivial privacy reasons, I would like to configure (opting-in/opting-out) the auto-play feature, with a general mute-all flag, and/or a bot-dependent flag (I probably want to un-mute just the more frequently used bots).

Related to the voice auto-play experience (and in general about any audio or music content play), of course a smart-phone has not the loudspeaker audio power of a smart-speaker (as the Amazon Echo or Google Nest devices, all great devices, no doubt).
And even if enhanced with a desired audio auto-play software feature, the coupling with an external (bluetooth-connected?) loudspeaker is something highly recommended, but optional.

Change request #2: 🎙️ Voice Wake-Word Detection

This feature is a bit more complex to implement, and controversial. You would replicate the user experience of the invocation sentence on a smart-speaker, when you say “Hey Google…”, sorry, I mean:

Hey Telegram open MyAppName…

You want the named bot starts waiting your spoken command utterance, recording your voice until a silence is detected, and afterward the audio message is forwarded to the Telegram server as usual (at the end of the day to the MyAppName bot).
There are also alternative solutions where we come to invocate a specific bot (by example you could want to invocate bot with his specific name, etc.), but you get the idea. Last but not least, the current Telegram client app push-to-talk chatbot selection mode is not so bad, eliminating many privacy-related concerns.

Change request #3: 🖧 A Decentralized Architecture Support for Voice-based Bots

Decentralized, headless or no-brokerage, are possible keywords that could differentiate Telegram bots from the centralized (1st party master-bot + 3rd party skill-bots) approach big-players so far imposed on us.
In these well-known scenarios the master company supplies a proprietary first-party master-bot (e.g. Google Assistant or Alexa) that commands all the games. In facts in this model, your bot (the action in Google Assistant parlance, or the skill in Amazon Alexa parlance, or the capsule in Samsung Bixby jargon) must follow a rigid contractual and technological framework and all data “passes through” central master-bot cloud servers. That’s bad, or at least, that’s controversial in terms of data privacy, vendor lock-in, etc.

So forget the first party approach of a (big-player owned) cloud system that monopolizes/centralizes traffic of all external (third-party) skill-bots. Instead, imagine Telegram just as a light middleware that supply two things:

• The common client (the TG app) updated with the above proposed change requests to integrate an enhanced voice integration
• Some server-side services on the Telegram cloud server that finally redirect to your (private person or enterprise) bot.

In other words, as final user:

Imagine you could access on your smartphone Telegram app a short-list of voicebot services provided by independent suppliers you (and only you) selected!

In this scenario Telegram would be not the unique (voice) assistant, but just a vehicle to enable you to “talk” with a set of independent assistants of your choice. It’s a complete different business model (with respect of big-players), requiring just minor architectural and software updates on the current Telegram client (and some optional enablers on Telegram servers).

Is this bot-independent model sustainable for Telegram company?

That’s a critical open point. I guess Telegram could set-up the overall architecture:

• Giving for free some basic features (by example the audio auto-play and the wake-word detection on the client, etc.)
• Supplying as paid services some server-side enablers, as a multilingual speech-to-text and text-to-speech platforms, and why not some smart “conversational AI” services to simplify/automatize chatbot developments (as a speech-to-text engine, a text-to-speech engine, an intent/entities classifier, a dialog manager, a semantic search engine, etc. etc.).

Maybe Telegram Premium, extended for a developers program (“Telegram Premium for Enterprises”), could be a possible solution to cover all costs of the upgrade and a possible economical gain?

I imagine by example that TG Premium for Entrprises contract could be set up, in such a way that a chatbot developer company could pay for premium services like a networking traffic enhancement, the use above mentioned server-side enablers, etc. etc.

Please comment!

What do you developers or final users think about all this?
Please leave your feedback on the comments!

Giorgio

Could Telegram be a competitor of voice assistants, like Amazon Alexa or Google Assistant? was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

Update: I participated at the www.lingofest2021.com event, 2021, March 26th, presenting my talk: Enterprise Voice Cobots, where this case study is deepened. Slides and video links available at the end of the article.

Voice assistants, for consumers at home, are nowadays taken for granted, but there is a huge space of applications of voice virtual assistants also in enterprise verticals.

I want to introduce my current R&D project, involving an innovative voice assistant application in the logistics shipping container operations.

In a sentence, the voice-cobot I conceived helps forklift vehicle drivers to load and unload shipping containers from/to yard spots from/to container trailers.

Let me share, before all, the general concept of a voice-cobot, as a possible application of a conversational assistive-reality computing. Afterward I will show the solution I found for the specific industrial scenario.

What’s a cobot?

On Wikipedia you get this definition:

“Cobots, or collaborative robots, are robots intended for direct human robot interaction within a shared space, or where humans and robots are in close proximity.”

Well, the above definition refers to industrial robots, hardware machines — capable of carrying out a complex series of actions automatically — guided by a human being, using an external control device or whit a control embedded within. Robots may be constructed on the lines of human form, but most robots are machines designed to perform a task with no regard to their aesthetics.

So in general we refer to cobots meaning industrial robots (the common case is an articulated electro-mechanical robots) that helps automate unergonomic tasks such as helping people moving heavy parts, or machine feeding or assembly operations. But that’s not what I want to talk to you about!

So what’s a Voice-Cobot?

For voice-cobot I mean a voice-interfaced digital assistant that, through a real-time spoken conversation, helps a human operator to accomplish a specific working task.

Now forget the usual virtual assistant scenario that, as end-users, we experiencing with Amazon Alexa or Google Assistant at home, through smartspeakers or smartdisplay devices. Instead I want to talk here about the special case of private virtual assistants for industry enterprises.

The disruption of a voice assistant in enterprise spaces refers to the concept of an assistive-reality computing automation made by a private, company-owned virtual assistant that collaborates with human operators (employees, skilled workers, professional technicians) to accomplish working tasks, literally having real-time conversational interactions (by voice, text, other UI).

Voice is an essential requirement, but the voice interface itself is not the game-changer, instead the collaborative enterprise-assistant computing is!

There is a common misconception where the novelty is just about the fact that you “talk to machine”, using speech (instead of chatting or using a graphical user interface), but that misses the most important point.

A voicefirst interface and maybe a voiceonly interface are — in many cases — the best way to interact with computer, in situations where the human is working “hands-on” as in the case of a vehicle driver, a machine operator, a doctor, etc.

Nevertheless, the best human-machine interface must be evaluated case-by-case and it can be made with different strategies, by example in input it could be voice, text, a camera scene, any IoT sensors, where in output it could be voice, earcons, text, graphics, light-signal devices, etc. All this could also work in parallel in a multimodal strategy.

The real innovation is not just the voice interface, but the virtual assistant collaborative logics, meaning that

enterprise processes are controlled by a single conversational-AI assisted-reality computing that interacts with human operators,

collaborating with them to accomplish workflows, and leading to time and costs savings and possibly to a better (and fun) user experience.

The case study: empty shipping containers handling

As a software engineer, consultant and researcher, I was initially asked by DITEN — Università di Genova- Dipartimento di Ingegneria Navale, Elettrica, Elettronica e delle Telecomunicazioni — to solve an apparently standard computer vision text detection/recognition problem.

The topic was to automatically recognize the shipping container marking codes, during the containers loading/unloading tasks made by an operator driving an empty container handler machine.

After some fun (I’m ironic) with text recognition from images algorithms I implemented, I soon realized how difficult is to detect texts in real-life motion scenes. It’s hard to achieve a detection system with an accuracy near to 100%. And what to do in case of the no detection/erroneous cases?

That’s why I got a trivial idea:

What if the operator dictated the code to a voice assistant? Just talking!.

My friend and account manager at DITEN University replied:

Let’s deepen, Giorgio, maybe it’s not crazy as it seems.

So Forklift-cobot born!
The concept was to supply to forklift vehicle operator, specifically empty container handler and reach stacker, a simple voice assistant command-and-control software, running on common tablet/mobile device fixed on the vehicle cabin.

The foreseen operator’s user experience concept is really simple: he gives voice commands (pull mode) to insert data, as the task name, the handled container code, the container-truck vehicle plate, yard spot name, etc.

The voice assistant checks dictated/spelled data, does search in the company backend database and stores transactions on task completion. Avoiding any stop of the operator just to insert data on a web GUI app (as currently happens).

I made two short video demonstrations of the proof-of-concept desktop-prototype I implemented, where I show the functionalities and the voice interaction activation on a mobile device.

I’m Italian and the the video are realized in Italian language, but you can enable Youtube subtitles in English.

Research & Development open points

The implemented prototype is now in the on-field test phase, where the proposed system has to be evaluated by expert senior machine operators.

Besides, there are many technical open points, mainly related to the voice recognition/audio subsystems and the on-cabin user interface ergonomics. Let me introduce some.

Speech recognition issues
Noise is a common industrial environment problem and an urgent related topic is the availability of a on-premise noise-proof ASR avoiding any cloud-enabled service. For security reasons, a key requirement is data-privacy: all business processes data transactions must not exit outside the enterprise intranet.

You need local and private (on-premise), multi-lingual, noise-proof robust speech recognition engine (ASR).

Voice activation UI, human-machine HW interfaces
A not-trivial aspect is related to find the suitable hardware interfaces with the cobot, case-by-case.

The current speech activation solution is done with multiple concurrent (working in parallel) push-to-talk solutions, as touchscreen, foot-switch activation, or physical push-button on the vehicle dashboard (see figure here below).

Also the mic/headphones subsystem has challenges solvable with many audio options: headset, open-air mic/loudspeakers, etc.

The webapp-based architecture
As shown in the videos, the prototype has been implemented with a client-server web architecture. Having implemented the client as an application running on a standard web browser on top of any mobile device, has many pros.

All the audio message exchange is realized with the Web Audio API and currently any web app (on a mobile device) can access also any internal devices (even the video camera streaming), the GPS geolocation coordinates and accelerometers (helping to localize the vehicle movement), even USB interfaced external devices (e.g. a RFID long-range reader), last but not least some Bluetooth interfaced peripherals (audio I/O, custom buttons), etc.

To wearable or not to wearable?
Another advantage of having the client running on a (web app) mobile device, is just the fact the “terminal” of the cobot run on a very cheap portable handset. You can use pretty any mobile phone or tablet, to be mounted inside a cabin and/or to be used outside the cabin, by an operator walking with the handset, for different tasks, as container positions control in the yard area, etc.

That said, the mobile device web client is just an option among many others. A “fixed” client could run on a more powerful edge-computer, or even a micro-controller.

Another alternative to mobile tablet/phones is a smartglasses wearable. Apparently that’s the definitive solution, but with a lot of issues like high costs, lack of standard API interfaces, unergonomic for an operator inside a cabin.

Conversational design
On the (software) user interface side there are interesting conversational design research topics. The current UI implementation uses a tablet/mobile device. The client-side software runs on a web browser and exploit some multimodal paradigms using voice, texts and, last but not least, usual graphical interface capabilities of a browser.

By example, in the prototype, the screen background color has been used to visually explicit the status of conversation turns between the user and the machine. Synthetic voice (TTS) short answers and prompts are accompanied by longer explanations written on the display, added by suggestions of the next steps the user could have to do, etc.

The command-and-control approach is less simple than what appears, especially if the conversation between user and the machine has to be at the same time short but nice and even engaging.

Maybe there is a challenge if you want to integrate the task-completion essential features (minimal requirements), with a domain-specific or open-domain question-answering query system, or you want to manage any user-defined alerts and reminds, an inter-operators intercom subsystem, etc.

Conversational AI backend computing
Here’s where it gets tricky. Consider the voice-cobot not just as yet another vertical application to solve a specific enterprise task.

Instead the cobot become the “enterprise’s computer” able to talk with the single user with possibly a user-defined botpersona, adapted to that user needs and preferences, and at the same time serving users for many requirements, as a single “company voice”.

The company cobot bot-persona appears to users as different (user-defined) bot-persona while being a single business logic intelligence.

Last but not least, the kernel component is the dialog manager / conversational AI intelligence.

For the prototype, because of the pretty “simple” scenario, I used my own dialog engine NaifJs — that I opensourced in 2020 — where dialog tracking is based on a state-machine approach.

For more complex scenarios, with many concurrent tasks and push mode/mixed-initiative where the assistant start a conversation, maybe to assign tasks, etc. the current implementation could be enhanced with an high-level rule-based language. This is an open research topic.

Another applied research/engineering topic is about how to “standardize” the company knowledge-base/database integration.

The problem could be solved trivially with APIs and database queries but the real challenge is to define a standard common ground company private knowledge base, to be queried and used for inferences made by the conversational AI.

Temporary conclusion

The described case study describe a very specific shipping industry scenario where a voice-cobot helps the working tasks of a container handler vehicle operator, reducing times and costs.

The described assistive-reality system is applied to a specific industrial operation, but it could be applied to many other factory tasks and different operator roles. By example, in the shipping container depots/repairs industry, the voice cobot could assists many other kind of human operator activities: truck gates automation, container inspection reporting, safety/emergency alerting, truck drivers “help desk”, operators tutoring, etc. etc.

Is all not just about another voice-interface application in industry. It’s instead about the rethinking of all business processes, where an assistive enterprise computing could collaborates with many/all humans.
Isn’t this just Industry 5.0?

2021, March 26th I participated at the www.lingofest2021.com event, presenting my talk: Enterprise Voice Cobots, where this case study is deepened.

Slides @ #lingofest2021:
https://docs.google.com/presentation/d/1ieZnAdREzEGXkcO4C_XPIbS9YAnE76mB0wpP2k-yOlQ/edit#slide=id.g4412d4946c_0_0

Video @ #lingofest2021:

Contact

If you are an enterprise company, maybe in the shipping / supply-chain/ smart-factory automation, or in any vertical where you think a voice-bot could solve a real workflow, or you are in R&D ICT company, or if you are an academic organization and you are interested on deepening this applied research contexts, I’m available to collaborate, as researcher and as a consultant.

You can contact me on linkedin or just send me an email at giorgio.robino@gmail.com.

Voice-cobots in industry. A case study was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

I learned of Almond, the Stanford University open virtual assistant, for the first time one year ago reading an article on voicebot.ai.
I have been immediately enthusiastic about the core concepts on which the project is based: the user’s data privacy, the need of a web based on linguistic user interfaces, the distributed computers architecture, the natural language programming approach and many others topics related to a possible next generation of the web, populated by a federation of humans and theirs personal assistants.

Soon I discovered that one of the principal developers of Almond software is Giovanni Campagna, a PhD student in the Computer Science Department at Stanford University and member of the Stanford Open Virtual Assistant Lab (OVAL) who works with prof. Monica Lam. Hence the idea of my interview with him, about the past and the future of Almond and personal virtual assistants.

Introduction to Almond

Giorgio: Ciao Giovanni! In order to sketch out what it is and what it will be Almond, may you briefly summarize the project history? The oldest public presentation I remember was by Monica Lam in fall 2018. May you give us the basic concepts of the project and explain what is the role of Almond within OVAL laboratory research?

Giovanni: The Almond project started in Spring of 2015 as a class project to explore a new, distributed approach to the popular IFTTT service. At the time, it was called Sabrina. Soon after, we realized the need for both a formal language to specify the capability of the assistants, and a natural language parser to go along with it, so end users could access those capabilities. The first publication for Almond, and the first reference under its current name, came in April 2017 at the WWW conference, where we described the architecture: the Almond assistant, the Thingpedia repository of knowledge, and the ThingTalk programming language connecting everything together.

Since then, we have been working on natural language understanding, focusing in particular with the problem of cost. State of the art NLU, used by Alexa and Google Assistant, requires a lot of annotated data, and is very expensive. We made two recent advancements: first, in PLDI 2019 we showed that using synthesized data can greatly reduce the cost of building the assistant for event-driven, IFTTT-style commands. Later, in CIKM this year my colleague showed how to build a Q&A agent that can understand complex questions over common domains (restaurants, hotels, people, music, movies, and books) with high accuracy at low cost.

In ACL this year, I presented how the same synthesized data approach can be used to build a multi-turn conversational interface, achieving state of the art zero-shot (no human-annotated training data) accuracy on the challenging MultiWOZ benchmark.

Most recently, we have received a grant from the Alfred P. Sloan Foundation to build a truly usable virtual assistant (not just a research prototype), and we hope to release it in 2021.

About Giovanni Campagna as researcher and developer

Giorgio: I’m curious about your personal history, Giovanni. I know you are Italian and probably you moved to Stanford as a PhD student. What made you start developing Almond? Did you initially work on Almond as part of your PhD thesis? Are you the leader of the software project development? And how is the team composed? Besides the Almond project, What is your current role in the OVAL team and what are your long-term research interests?

Giovanni: I moved to the US and to Stanford as a Master’s student, and I started Almond in class. I was interested in programming languages and pursuing the software theory concentration, which includes PL, compilers, and formal methods. I met Prof. Lam in her Advanced Compilers class. At the time, she was mainly focusing on messaging and social networks, with the goal of disrupting the Facebook monopoly (this was part of the Programmable Open Mobile Internet initiative, aka the Mobisocial Lab). But even then, she had the vision of what would come next, and while Alexa was not popular yet, it was clear that virtual assistants would be the next potential monopoly.

Personally, I started developing Almond as research on the ThingTalk programming language: designing a programming language that can enrich the power of the assistant beyond simple commands, and give the power of programming to everyone. Parallel to ThingTalk, I worked on converting natural language to code, because natural language is the medium of choice to make programming accessible. Our research found that neural networks are extremely effective for natural language programming tasks, as long as training data is available. ThingTalk is still the core of my PhD thesis, but over time we moved our focus to reducing the cost of training data acquisition.

I built the original version of the Almond assistant, and a large chunk of the current code, so in a way I am still the maintainer of it, but Almond would not have been possible without the help of my colleagues. These include Silei, Michael, Jackie, Mehrad, and Sina (all current PhD students). Silei was the first to join after me and he’s the second-most active dev on Almond. His research is mainly on Q&A over structured data. Michael recently defended his thesis; he was working on multi-modality: bridging GUIs and natural language. Jackie also worked on multi-modality, with a paper in UIST on mobile app interactions using natural language. Mehrad is working on multilinguality leveraging machine translation technology, as well as named entity recognition. Sina is working on Q&A over free text, paraphrasing, and error correction. Additionally, we have a number of MS and undergraduate students who have helped on various projects.

What is your view on data privacy and how this is related to personal assistants?

Giorgio: Data privacy is probably a foundation concept that gave birth to the Almond project. May you deepen why privacy is so important for all of us, citizens and companies? How is all this related to democracy and people’s freedom on the web?

One problem I see in current big player personal assistants is the fact that people’s data (voice conversations, also containing background ambient audio, e.g. at home) are processed by cloud systems proprietary platforms. In the best scenario, all this data is used to improve “AI-blackbox” cloud-based proprietary services, feeding machine learning algorithms. In the worst case, conspiracy theorists do suppose a malicious use by such companies that would steal personal end-user data populating people’s knowledge bases for any further commercial usage. Do you consider this last scenario an actual concern?
To protect privacy, in opposition to cloud-based “walled gardens”, Almond provides a tech architecture based on virtual assistants that can run on users’ devices. Do you think that the issue can be fully solved at the architectural level or do we need in any case government regulations?

Giovanni: This is not an easy question, and let me preface by saying this is my personal opinion, not the project’s. I think privacy is inextricable from freedom: I am not truly free if everything I do is tracked, logged, and stored forever by a company or government. I am not truly free if I can be judged in the future for anything I’ve done at any point in my life. One closes the blinds to be free to do whatever in the privacy of their home. And because so much of our lives is now conducted over the Internet, it’s clear that Internet privacy overlaps significantly with real life privacy.

Now, as you point out, virtual assistants and conversational AIs in general pose unique challenges to the privacy problem. First, state of the art natural language requires a lot of user data for training, which means the virtual assistant providers are continuously collecting all the conversations performed on the assistant, and have contractors continuously listening and annotating the data. To have somebody listen to my conversations, that’s not very private. Reducing the need for annotated real data has been a strong focus of our research, and we believe we’re finally getting there.

Second, and most importantly, virtual assistants inherently have access to all our data, through our accounts: banking, health, IoT, etc.. We want the virtual assistant to get access to our accounts because we want help, and we want the convenience of natural language. But what guarantee do we have that a proprietary service won’t suck all the data and use it for marketing purposes? Why wouldn’t a proprietary assistant provider look at our banking information to promote a credit card or mortgage product? Why wouldn’t a proprietary assistant look at the configured IoT devices to promote similar or compatible products? And with Amazon and Google dominating the online retail and ad markets respectively, it would be surprising if they did not eventually start doing that.

Tell me about your general vision on open-source software

Giorgio: I see that many Almond software components are made in Node.js and I know you have been a member of the Linux/GNOME community. Could you share your point of view about the importance of open-source software in general, such as the Linux operating system (I know you are a Linux desktop user, like me). How open source and open data are related to data privacy?

Giovanni: I have been an advocate of free software for a long time, and I am a strong believer of the four fundamental software freedoms as purely ethical principles. The freedom to study software is what allowed me to learn how to build software before I started college, and the freedom to modify and distribute software allows people to collaborate and build something bigger than any single individual could build.

I also believe that, unlike proprietary software, free software cannot abuse the trust of users. It is trivial to detect a free software app collecting more data that claims, or doing anything shady with the data. It is trivial to fork the app and remove any privacy-invasive functionality. Hence, free software communities are very careful to gain the trust of users and protect their privacy. You can see it for example with Firefox: while Firefox collects data for telemetry, they’re very careful to allow people to disable the telemetry, and do not collect more than they claim.

Giorgio: Why did you decide to develop Almond in Node.js? As nodejs developer myself, I’m specifically curious about the engineering reasons that drove you to use the Javascript environment. Is it an opportunistic matter, maybe because it is easy with nodejs to develop in multi-platform environments? Or are there other software engineering reasons?

Giovanni: I chose to build Almond in nodejs because it is the most portable platform. At the beginning, we had this idea that the full Almond assistant could run on web, phone (Android and iOS), desktop, embedded. Nodejs is the only platform that supports that. Over time, we found that running the assistant on Android or iOS is quite challenging, and we moved to an architecture where a user keeps the assistant running on a home server or a smart speaker. Yet, I still find nodejs more programmer friendly and just nicer to work with than the obvious alternative, Python. I should also note, using a type-safe compiled language would have been challenging, given the ever changing nature of a research prototype.

What do you mean by Linguistic Web?

Giorgio: Regarding Almond’s core concepts, I have been impressed by the statement “We are witnessing the start of proprietary linguistic webs” that I found in these Almond presentation slides. May you clarify what you mean by Linguistic Web? Are proprietary linguistic webs an implicit reference to Google Assistant and Amazon Alexa voice-based / smartspeaker-based virtual assistants? If so, what are your concerns in the “walled gardens” proprietary (virtual assistants) ecosystems and how Almond could be an alternative for private citizens and/or companies? What are strengths of open and non-proprietary platforms to improve (linguistic) democracy (and freedom) on the web?

Giovanni: That is exactly right: what we’re referring to is the third-party skill platforms being walled gardens controlled top-down by the assistant providers. Any company who wishes to have a voice interface must submit to Alexa and Google Assistant. As proprietary systems, these can shutdown competing services, or impose untenable fees.

We believe instead that every company should be able to build their own natural language interface, without depending on Amazon or Google. These natural language interfaces should be accessible to any assistant. One example of work in this direction is Schema2QA (to appear in CIKM 2020), a tool to build Q&A agents using the standard Schema.org markup. Any website can include the appropriate markup to build a custom Q&A skill for themselves, and furthermore the data is available to be aggregated across websites by the assistant.

Could you explain what Natural Language Programming is?

Giorgio: Could you define what LUI (Linguistic User Interfaces) and Natural Language Programming mean for you? With the popularity of smart speakers and current voice-first interfaces we are moving from the GUI (Graphical user Interface) to CUI/VUI (Conversational / Voice User Interfaces). One of Almond’s disruptive milestones is the idea that the final user must program himself his personal virtual assistant, just speaking with a computer in natural language. That’s currently an unachieved goal, in human-machine interfaces.

What do you think it will be, say in the next ten years, the way users will program their private virtual assistants? More in general, do you foresee that common people will interact with computers (and professional software developers will develop applications) in some sort of natural language programming?

Giovanni: To me, there is no distinction between LUI, CUI and VUI. Of course, language is conversational, and we expect the assistant to sustain multi-turn conversations, with follow-ups and error correction. This is by the way something that Alexa and Google do very well for their first party skills, but don’t really offer to their third-party skills, which get basic single-shot intent classification and perhaps simple slot filling. (In contrast, Bixby is another assistant that is built conversational from the start, and in many ways has a similar design to Almond). Note also that voice tech is mature and standard STT works really well.

Where things change is natural language programming. Ultimately, the goal of a virtual assistant is to use natural language to do things, and because the assistant is a machine, all it can do is execute code. So the idea is that every natural language command issued to the assistant can be mapped to an executable statement in a programming language (a domain-specific language, in our case ThingTalk). The job of the assistant then is just to execute the generated code and present the results to the user. Once you frame the assistant this way, the capability of the assistant is only limited by what the DSL can represent. For example, we experimented with a DSL of access control policies, and found it could be used to grant fine-grain access to shared devices and accounts (in Ubicomp 2018).

Which is the personal assistant end user experience, according to Almond vision?

Giorgio: Which kind of personal assistant user experience Almond provides to end users? It seems to me that Almond is built upon the task-oriented approach to automate everyday user actions/tasks (especially browsing/querying the web), following the IFTTT-like way (“alert me when the price of BitCoin is below $3600”). This approach reminds me of the Google Assistant mantra “To get things done” and that’s smart! This kind of personal-programmed micro-tasks completion feature is in fact pretty absent (or just hinted) in big players platforms such as Google Assistant and Amazon Alexa.

On the other hand, Google and Amazon provide some sort of general-purpose / not-personal basic question-answering and (news/ music) streaming services, referring to third party developers (Action in Google parlance / skills in Amazon parlance) for any other specific service.

May you deepen the key values and UX features that differentiate Almond from big players?

Giovanni: First of all, I want to stress, Almond is still a research prototype. It is an experimental platform to test our ideas, both in NLP and in HCI. We have received a grant from Sloan to turn the prototype into a truly usable product. As we do that, we imagine we will also focus on the most important skills: music, news, Q&A, weather, timers, etc. Yet, the technical foundation to support end-user programming will remain there, and we will try to support it going forward.

In terms of differentiating features, I imagine the key differentiator is really privacy, rather than UX. I imagine Almond will be supported on a traditional smart speaker interface, because that’s the most common use case for a voice interface. I also personally like using Almond on the PC, where we have an opportunity on the free software OSes. I’ve given a talk at GUADEC (the GNOME conference) recently about potential opportunities there.

Giorgio: What do you think about the big player server-centric (1st + 3rd party) information architecture, especially in terms of quality of service to end users?

Giovanni: Finally, because we’re fully open source, I think the distinction between first party and third party will be blurred in our assistant. We give the same technology to everyone, unlike Alexa for example which keeps AMRL (the Alexa Meaning Representation Language) only for first party skills. We imagine that even long tail skills will be developed in an open repository, and everyone will collaborate to build those skills.
The model should be similar to Home Assistant, a leader in the open-source IoT space, which is all built by the community.

About the NLP chain: LUINet, ThingTalk and Thinkpedia

Giorgio: Could you introduce the Almond natural language processing chain you conceived?

Giovanni: The key idea of our approach to NLU is to factor the domain-independent aspects of natural language from the specific domains. Our goal is to raise the level of abstraction, so that developers don’t have to build the same thing over and over again. Instead, we want developers to specify their APIs and database schemas, with a few bits of natural language on every field. We then use a general state machine of dialogues and a general grammar of English to synthesize millions of dialogues that talk about the domain of interest, which we train on. The tools to build these synthesized datasets for training are part of Genie, which is the core NLP technology backing Almond. In diagram form:

At inference time, this is what the agent does:

Our pipeline uses a neural semantic parser (the LUINet model, based on the BERT-LSTM architecture) to understand the input sentence, and maps it to an executable form in the ThingTalk programming language. The ThingTalk code makes use of primitive APIs defined in Thingpedia, such as the Yelp skill in this example. The code is JIT compiled and executed, and returns the results. The results are then passed to a general dialogue state machine, that, given the AST of the executed code, the results, and annotations on the APIs, is able to generate both the new formal representation of the dialogue, and the agent’s utterance.

Here is the state machine at a glance:

The interesting aspects of this state machine is that it does not depend on the particular domain of interest. So the same state machine can be used for restaurants, movies, music, etc. The state machine is built once and for all, and new domains can be plugged with very little cost. Additionally, when the state machine is refined to add new features, the refinements are shared across all skills. This is also a way in which all skills are “first-party skills”: all skills benefit from the work done to improve other skills, when that work is not domain-specific.

About the technology behind the Almond NLP chain

Giorgio: The work you made with LUINet, Genie, and all other components is impressive! It seems to me that you followed the “classic” semantic parsing approach where natural language statements are translated to a formal language (ThingTalk). The semantic parsing approach makes absolute sense for me, maybe differentiating itself from currently very popular intent-based probabilistic classifiers approach, used by Google (Dialogflow), Amazon (Lex) and many other NLU (Natural Language “Understanding”) platforms available on the market.

Could you deepen how the Almond approach differs from the intent-based classifiers? What are pros and cons of these different approaches?
Don’t you think that in the long term the intent-based machine learning could be a simpler way to let the assistant learn (”retrain”) new user intents/requests? On the other hand, in the long term, the Almond NLP formal language production would win in terms of AI explainability in a possible “machine reasoning”. What do you think about it?

Giovanni: First, let me clarify that when we say semantic parsing, we mean neural semantic parsing, rather than classic semantic parsing. Classic semantic parsing is template or grammar based, and tries to match spans of the sentence to specific primitives in the knowledge base.
Neural semantic parsing instead has a lot more in common with machine translation: a sentence is fed to a neural network, and the neural network outputs a program, token by token.
That makes neural semantic parsing a super set of intent-and-slot systems: intent-and-slot are similar to semantic parsing, where the target language has a single function call with parameters.
The neural semantic parsing approach is more general, because the target language can be any formal language, and need not match the sentence exactly, nor it needs to be limited to only one API call. For example, semantic parsing allows us to translate questions to SQL-like statements with joins, projections and filters, instead of hard-coded API calls, which allows us to better understand complex questions.

The downside is that neural semantic parsing requires more data to train, which, if annotated by hand, must be annotated by an expert. Building a semantic parsing training set by hand is practically infeasible: the closest that has been built are dialogue state tracking datasets such as MultiWOZ (which are known to have annotation problems) or paraphrase-based datasets such as Overnight, WikiSQL and Schema-Guided Dialogues (but performance on paraphrase is known to overestimate performance on real data). On the other hand, Genie allows us to use synthesis for the training set, and only annotate a small amount of data for evaluation, which makes semantic parsing practical again.

I imagine long-term all assistants will move to semantic parsing. Note for example that Alexa is also using some form of semantic parsing (through AMRL) for first-party skills. Intent classification is too limited in what kind of sentences are understood.

Genie also differs from other assistants in how it approaches state tracking (carrying over state across multiple turns). While commonly used tech separates NLU and state tracking, Genie combines both problems into a single neural network. This reduces the problem of “unhappy paths” typical of rule-based state tracking, where all the different ways the user might continue the conversation or change the subject have to be modeled explicitly.
The rule-based state machine is still imbued into the neural model using the state-machine based synthesis, but the neural network can generalize beyond it. In our experiments on the MultiWOZ dataset, we found the state machine could cover 83% of turns, but the neural network could still interpret correctly 47% of the remaining 17%, hence generalizing in what state transitions are supported by the agent.
See paper: State-Machine-Based Dialogue Agents with Few-Shot Contextual Semantic Parsers.

Will ThingTalk evolve from a smart commands interpreter to a full “conversational companion”?

Giorgio: I see ThingTalk, the Almond natural language programming language inspired by IFTTT, as the first attempt to implement a general natural language programming. Using ThingTalk you can set-up in natural language some “actions’’ triggered by external (web APIs/local) events (e.g. “When I use my inhaler, get my GPS location, if it is not home, write it to a log file in Box.”).
Does ThingTalk also include any personal-facts continuous learning and a personal knowledge base memory?
Do you have any news about how Almond could eventually evolve into a general-purpose personal “conversational AI”, able to sustain multi-turn conversations, not only in event-based task-completion contexts, but maybe also in companion-like open-domains / chit-chat dialogs?

Giovanni: This is absolutely on our radar. First of all, we’re partnering with Chirpy Cardinal, another Stanford project who won second place in the Alexa Social Bot Challenge. In the near future, we will integrate Chirpy Cardinal into Almond for companionship and chit-chat capabilities.

We also imagine that the assistant will learn the profile of the user, their preferences, and will have memory of all transactions, inside and outside the agent. We do not have any released work on this yet.

About stateful and contextual dialog management

Giorgio: In general, one topic I’m personally obsessed with is how to program multi-turn chatbot dialogues, in contextual (closed/open) domains. That’s a goal not yet achieved by Google and Amazon cloud-based assistants. To date, in fact, both the famous systems surprisingly do not maintain dialog context in multi-turn conversations even on a simple domain as weather forecasts. The lack of context is not just related to conversational domain, but also to the “time”. Namely, the above mentioned voice assistants are not able to remember pretty anything about a previous interaction with a specific user. No memory (“stateless”, if we think about a conversation as a state-machine workflow). Worst, there isn’t any incremental learning by conversations.

Now, in what directions do you think conversational technology will evolve? Personally, I foresee a next generation of personal assistants that will be able to sustain task-based / closed-domains dialogs (say in the ThingTalk way) and to chat about general open-domain knowledge. The basic personal assistants feature I do not see yet in any state of the art chatbot is the ability to understand and reason about personal user facts. Do you agree with this view?

Giovanni: I think state of the art assistants will grow conversational capabilities for task-oriented skills very quickly. Some, like Almond and Bixby, are built to support multi-turn from the start. Others, like Alexa, will require re-engineering for multi-turn, but they will get there very soon. See also Alexa Conversations as an emerging technology for multi-turn, multi-skill experiences.

Incremental learning is a much more open-ended area. There is a large body of work in this space, starting with LIA, the “teachable” assistant from CMU. I also imagine the assistant will grow a profile of the user, both by data mining on the conversation history and by explicitly tracking a KB of the user’s information. In a sense, this is already available: the assistant knows my contacts and family relations, it knows my location, it knows my preferred music provider, etc. It will only grow over time as more features are added.

Giorgio: BTW, what is your opinion about any practical usage of “statistical web-crowd-sourcing” (my definition) in systems like Generative Pre-trained Transformer 3 (GPT-3), the auto-regressive language model that uses deep learning to produce human-like text?

Giovanni: Pretraining is at the core of the modern NLP pipeline, whether it’s masked-language-model “fill in the blanks” pretraining (BERT and subsequent works), generative pretraining (GPT 1, 2 and 3) or sequence-to-sequence (T5, BART). It is key to understand language, because it can be trained unsupervised, so it has significantly less cost than supervised training. I can only imagine the use of pretraining will grow over time. As for GPT3 specifically, the few-shot results are honestly impressive on a range of tasks. At the same time, the model is so large that it cannot be easily fine-tuned, so it’s quite difficult to apply it to a downstream task.

Giorgio: About closed domain vs open domain chatbot building, what do you think about RASA (the open-source opensource engine to build contextual assistants) approach?

Giovanni: What RASA is doing is quite interesting in that they’re also trying to push the envelope of conversationality, and they also recognize the limit of intent-based systems. At the same time, I see their current NLU product is still using a classic intent-based dialogue tree. Their dialogue manager requires fully annotated examples of conversations, which are incredibly hard to acquire and annotate well. But I’m looking forward to new stuff, when it becomes available!

The Distributed ThingTalk Protocol and the Federated Virtual Assistants Architecture

Giorgio: One of the things I love more of Almond is the vision of a next generation web made by a network of federated (Almond based) virtual assistants. As far as I understood, in this model each person would have a virtual assistant acting as a virtual secretary and talking with other people-clones assistants or people directly. The virtual assistant would act as a “programmable interface”, managing “access control” and sharing personal info based on dynamic programming made by users themselves. That is, in my opinion, very powerful and disruptive! May you explain this concept and could you give us some technical implementation architectural details?

Giovanni: I think your question summarizes it very well. The idea is that every person would have their personal virtual assistant running on their own trusted device. The virtual assistant executes requests on behalf of the owner, and on behalf of others, with access control. The requests are represented in ThingTalk and exchanged over a messaging protocol; in our prototype, we used the Matrix messaging protocol. The access control policies are also represented in ThingTalk. Access control is enforced using Satisfiability Modulo Theory, so the access control is formally verified. I recommend looking at our Ubicomp 18 paper for further technical details.
The interesting thing of this work though was noting how useful fine-grained access control would be: in our user study, we found that across 20 scenarios, the willingness to share data and accounts would double with fine-grained control. We also found that our access control language covered 90% of the enforceable use cases suggested by crowdworkers.

On which devices Almond will run: smartphones, smart-speakers, personal computers?

Giorgio: I know you spent a lot of energy trying to run Almond on a vast range of personal computer platforms, focusing on the Android app as a common “personal computer”, maybe because smartphones are the personal computers in this era, for common people. Beside, one of the possible weaknesses I see in Almond is the absence of a (home-based) voice-based interface, maybe through an open-hardware smart-speaker? Do you have any plan to allow private citizens to interface Almond through smart-speaker or any voice-based platform? What are pros and cons of voice-first interfaces?

Giovanni: We absolutely see Almond on the smart speaker as a first class citizen. Since fall of 2019 we have partnered with Home Assistant to bundle Almond as an official add-on, so you can use Almond to control a Home Assistant-based smart speaker. That means one can build a fully open-source voice assistant stack using a Raspberry Pi, Home Assistant OS, and Almond. There are a couple challenges in using Almond with a pure voice interface, mainly around the wake word, for which there is no easy to use open-source solution. (Recently, we discovered Howl from UWaterloo, which is also used by Firefox Voice, and we’re investigating that). Also, building a conversational interface that is friendly to pure speech output is not easy. Even commercial assistants work better on a phone when they can display links, cards, and interactive interfaces.

Could the Almond federated architecture be also a solution for business companies?

Giorgio: A distributed architecture of virtual assistants (where each end-user has his local assistant) that allows people to tune a fine-grained access control, selecting what info is public and what actions external assistants (aka people) can access to, seems to me a breakthrough in the current debate on personal data sharing.
BTW, You may know that Tim Berners-Lee in 2018 announced to be working on a personal assistant (code-name: Charlie). “Unlike with Alexa, on Charlie people would own all their data”. Are you in touch with him or any people at Inrupt?

Giovanni: I know that Monica has spoken with Tim Berners-Lee in the past. In any case, I believe this space is quite young, and there is certainly an opportunity for multiple open-source projects who focus on different aspects of the stack. Our focus is really in the NLP and dialogue management, while their focus seems to be the distributed architecture.

Giorgio: Almond seems now focused on providing an assistant to private citizens (end private users). The distributed architecture and the access control management you propose for end private users couldn’t be applicable also to business companies that want to provide their services to people? I imagine a scenario where an end user’s assistant talks to a company-assistant. May this Almond possible extension in the future be coupled with Thingpedia APIs?

Giovanni: Of course! The goal of our research prototype of a distributed virtual assistant was to show how useful access control can be in natural language. The use cases need not be limited to consumer access control: it could be applied in corporate settings, and it could be applied to sharing data between consumers and businesses. For an example of the latter, see this paper from HTC & NTU which uses ThingTalk technology to audit sharing of medical data.

First European Open Virtual Assistant Workshop

Giorgio: In June 2020, the First European Open Virtual Assistant Workshop scheduled by OVAL was cancelled due to the COVID-19 outbreak. The goal of the workshop was to introduce OVAL lab’s open, federated and privacy-preserving virtual assistant to the European research and business communities. There is any plan to reschedule the workshop?

Giovanni: Unfortunately, as you can imagine all in-person events have been canceled for the foreseeable future due to COVID-19. I don’t know at this time when the workshop will be rescheduled.

Giorgio: In general, what do you think about the recent European citizens’ privacy-preserving initiatives and related supporting laws (see GDPR regulations, the recent GAIA-X project and, specifically regarding personal assistants, the Fraunhofer SPEAKER platform)? Do you see common points between current European policy on “data sovereignty” and the Almond goals?

Giovanni: I think European efforts in this space are very important in terms of raising awareness of the importance of privacy. Building effective alternatives to the Amazon / Google duopoly is one way to restore privacy, like we’re doing with Almond.

At Almond, we’re also collaborating with the AI4EU initiative, which aims to build an European cloud of AI infrastructure.

At the same time, on a purely personal basis, as an European citizen with an opinion, I often disagree with the choices of our Commission, which seems to be animated more by economic strategy (and fear of American competition) than by sincere values. The goal should be privacy for all, not just making sure the next Google pays European taxes.

Often, it is also difficult to assess certain projects, because there is no open-source code, no released product, not even a development version. We see a lot of press releases and reports, but there is no sense of a coherent software artifact, accessible to developers. Even for the two projects you linked, development has reportedly started, but there is no code accessible anywhere. To me, and I stress this is a purely personal opinion, this is not the way to run a successful open source initiative.

How do you see the future of Almond?

Giorgio: Recently Almond received Sloan Support. As far as I understand, the new funds will support the engineering of Almond solutions, with the goal to convert the developed prototypes into real products usable by consumers.

May you give more details and describe the next steps of the project, in the short term and in long-term (say next 5 years)?

Giovanni: The short term goal, in the next year, is to use the funds from Sloan and other foundations to build an initial product. We aim for a small initial user base of enthusiasts who care about privacy.

The long term goal is then to use this initial product to further fund raise, and then use the established product to build both a successful open-source community, and an ecosystem of companies using Almond technology in their products. This should allow Almond to thrive and become self-sustainable.

Recent Talks (2020)

Almond: An Open, Programmable Virtual Assistant
Giovanni Campagna, GUADEC, the GNOME Conference, ORBIS 2020,
July 24, 2020.

Building the Smartest and Open Virtual Assistant to Protect Privacy
Monica Lam, Stanford Online Seminar, April 9, 2020.

Links

Giovanni Campagna home
Almond project home
Stanford Open Virtual Assistant Lab
Open Virtual Assistant Initiative Wiki
Almond Community Forum
Almond open-source repositories

How to contribute to Almond

Updates:
september 22th 2020: in paragraph “About the technology behind the Almond NLP chain” the answer has been integrated, inserting the link of a new paper.

Whither Almond, the Stanford University open virtual assistant, will go? was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some reasons why I’m developing a Telegram chatbot, giving up to develop an Alexa or Google Assistant application.

This article is a remake of my original answer to Julie Daniel Davis article Google Assistant versus Amazon Alexa: Which Could be Queen of the Classroom?
I mitigated the original title “Are Alexa and Google Assistant both looser in edutech space?”, too extreme and generic, I admit.
The reviewed article is above all a way of reflecting about my experience with CPIAbot, a chatbot I’m developing to assist no-native italian language L2/PreA1 (almost illiterate) students of some Italian public schools, as part of a ITD-CNR research project.
More broadly, my thoughts are about some limits of today Amazon and Google voice assistant and smartspeakers technology.

Almost one year ago I started to develop CPIAbot, a language-first, voice-first multimodal chatbot, running on Telegram, to assist foreigners, students of CPIA, Italian public adult schools courses, to learn Italian language basics.

Let me tell the story. In fall 2018, our initial research goal, at ITD-CNR, was to realize a smartspeaker application (on Alexa or on GoogleAssistant) but we faced many issues. I detailed a long list of points in an academic paper I’ve just submitted (soon available), entitled “Un assistente conversazionale a supporto dell’apprendimento dell’italiano L2 per migranti: CPIAbot” (more details at the end of this article).

Long story, short, about using smartspeakers in our ongoing experiment:

From the linguistics/educational perspective, a voiceonly application is too demanding in term of learner’s cognitive effort, especially in the case of no native/illiterate learner (QCER level L2/Pre A1) .

Unique User identification

There are also many related tech issues, but for me, in educational realms (in classroom or outside the classroom),

the big issue that both systems have is the lack of a real Unique User Identification.

That means a way to identify uniquely the user (any student, any teacher) of the (conversational/voice) application. Having an user ID requirement is for me essential for any chat or voice application made with the goal of follow the student learning.

The related big point is related to the impossibility to process voice recordings of students. In short:

an Alexa Skill or a Google Assistant Conversational Action can not access user voices. Full stop.

Here below I deepen the voice/text flow that happens when a user interact with a third party app on Alexa or Google Assistant (through a smartspeaker).

Personal Voice Identification

Voice signature identification is just a subtopic of users (students) identification need. Probably we could renounce to identify a student by his/her voice, but at the end of the day we need to identify the student (especially for personalized exercises, part of our application/research goal). If we want to track and improve the specific student learning progress, we need to identify him/her.

So far, both Alexa and Google Assistant do not provide a convincing/final solution to identify speaking user in front of a smartspeaker (a far-field device).

Giorgio Robino on Twitter

@4ICT Well, I'm discovering that: 1/ GoogleHome has "voice match" feature that could allow to identify all users did the matching procedure. Afterward GA could discard "remote waking", but that seems NOT possible. See: https://t.co/N8rxrLAnkx => functional privacy bug immo cont

Both big players allow to identify a small set of voiceprints (Google call these voice matches, whereas Amazon call these voice profiles). Google Assitant recognize a max of six different voices, whereas is not clear if there is a limit of recognized voices in Amazon Alexa. The good news is that both systems pass info to third party skills. See here and here.

But the proposed solutions are unpractical or even impossible in classroom scenarios, where there could be a lot of students in front to a single smartspeaker. All in all, the present voice recognition is not a suitable solution to identify a student.

Users Speech (voice recordings)

Voice recognition a part, there are other basic limits about voice recordings in general. The biggest issue involves privacy topics, but let take a part that thin ice for now; just let’s talk about some flat technical points:

both Alexa and Google Assistant “by design” do not forward the users (students in our case) voice recordings to third party skills/actions.

Generally speaking, this is quite understandable because big players do not want any possible malicious use of people voices by third party applications. But this inhibit a lot of smart elaborations the application could do with voice analysis. I already point out the need to use voice print recognition to identify speakers, and…

without the student audio/voice recordings, the application can’t do correct pronunciation analysis, sentence intonation recognition, emotional detection (“sentiment analysis”), etc. etc.

On Alexa, user utterances are not forwarded to the third party skills

In case of Alexa, user utterances (transcripted from voice to text) are not forwarded to third party skills, “as is”. In facts it happens that:

An Alexa skill do not even receive the full sentence (as voice-to-text transcript) of the user speech!

Instead, the skill gets just a label (an intent, in “conversational AI” jargon) that the developer has initially planned (during the “Alexa Skill nteraction model” design phase) that matches with the current user sentence. Strange, but this is the way!

Why Alexa do this “censorship”?

As far as I know, it has never been officially explained by Amazon, but I do believe it was a deliberate strategic decision. My suggestion is that the Alexa “interaction model” mitigates possible malicious uses/abuses by third party skills, giving Amazon an automatable way to control 3rd Party apps, avoiding privacy issues, etc. That’s a possible “customer first” right dogma.

On the other hand, the intent-based interaction inhibit third party skills to elaborate the user full utterance, limiting the NLU (natural language understanding).

Giorgio Robino on Twitter

new episode: to filter (Alex with "interaction model") or not to filter (Assistant Actions SDK) user utterances? Details: https://t.co/UTpk1tYa6j Google win this play.

Google Assistant, with Actions SDK, gives developers more freedom allowing them to use a classic intent-based classifier (the Dialogflow platform) as Alexa does, but Google also gives the option to use an alternative “low level” pass-trough (Actions SDK API), where the user utterances are passed to application, without any filtering. Thanks Google for that!

Why this “pass-through” is so important for an e-learning app or any (language) assistant bot?

Having the complete student text and voice input is paramount to analyse utterances on a human-machine conversation, e.g. a linguistic exercise.

Let’s imagine a simple conversation where an assistant chatbot asks the student to describe a scene displayed on a image or a video. In the example below, the CPIAbot exercise “guess the word” asks the student to guess the word (part of a glossary) describing the people that work in the scene.

Regarding the specific image, the students could answer: sales girls, women, cashiers, shop girls, clerks and many other definitions, that would be interesting to catch and analyze by the bot. This is feasible but pretty hard to implement with the Alexa interaction model.

Devices costs

The other point the original article mentioned are smartspeakers devices end-user costs. The author is absolutely right when she says that an Amazon Alexa Dot is cheaper than a Google Nest Home. The same is for Echobuds (Amazon offers the cheapest earables devices in the market). BTW, earbuds (personal, near-field earbuds) solve the user identification problem previous mentioned.

In a language learning context (and in any discipline), we want students use the (voice bot) application also outside the classroom, and

the “cheapest” and “easiest to own” device for people, especially refugees is a smartphone.

So back to our comparison, again both Google Assistant And Alexa lose in facts, because it’s true that both assistants are available as mobile app, but there are limits on voice/text interactions, by example in case of Amazon Assistant the students can’t text to Alexa. Google Assistant is a bit better, allowing users to write (or speak), but (if I well remember) there is no the way, action application side, to distinguish if user has wrote or has spoken (I could be wrong, I have to double check).

Minor notes about the devices audio quality

Amazon Echo has a jack audio output and Google Nest has not. That’s true. But both the devices could be coupled via bluetooth. Problem solved.

For me, as an “audiophile”, in terms of audio quality, comparing my Google Home Mini with an Amazon Echo Dot (v3), I much prefer the Google’s device because the more natural sound, or vice-versa, I literally dislike the over-compressed audio quality of Amazon Echo devices. Well, that’s subjective and on the other hand, in a classroom, the Echo’s bigger audio is a plus, I admit.

Authoring tools for teachers

Alexa Skills Blueprint? Maybe they are a nice tool for an initial engagement and gaming, but you can’t develop interesting/serious educational applications with them.

The most important point for me, is that both Google and Amazon do not provide so far a real simple convincing tools for non-developer and application developers in education realms, as teachers.

The visual design (GUI) vs language-based (CUI) tools is an old fashioned but highly discussed debate among conversational designers/developers. My side, regarding skill/actions development, and more in general about conversational application design and development, I’m supporter, since few years ago, of not-visual, but high-level authoring tools (possibly using very simple, declarative, ~natural language) programming languages. So far, I do believe that

both Amazon and Google do not provide authoring tools that allow teachers, contents creators and any no-developer, to easily create serious/complex custom applications.

Maybe Amazon is in the right direction with the www.litexa.com approach (I’ll deepen that topic on a future article). On the other hand, I have to say, as developer, that Google Actions programming paradigm is better than the Amazon skills programming proposal. There are many reasons, a bit tech/off topic here, that I often pointed out with my tweets.

Concluding, I confess I’m not too happy about how the two biggest players now support application development (in educational realms). We need much more! :-)

CPIAbot academic papers:

1. F. Ravicchio, G. Robino, G. Trentin.
CPIAbot: un chatbot nell’insegnamento dell’Italiano L2 per stranieri. 2019.
Published in Didamatica 2019 acts, Best Paper Award in section: BYOD. Mobile e Mixed Learning (ISBN 978–88–98091–50–8 https://www.aicanet.it/didamatica2019/atti-2019 p. 77–86).

2. F. Ravicchio, G. Robino, S. Torsani, G. Trentin.
Un assistente conversazionale a supporto dell’apprendimento dell’italiano L2 per migranti: CPIAbot. Nov 2019.
Submitted to the Italian Journal of Educational Technology (IJET).

Related article: Stateful Alexa Skills?

I’m happy to read your opinion. Please let me know your experience!

Are Alexa and Google Assistant both unfit as language learning assistant, inside/outside the… was originally published in ConvComp.it on Medium, where people are continuing the conversation by highlighting and responding to this story.