Introducing Libqmycroft

Libqmycroft is an Mycroft integration library using a mock skill interface API to integrate and register Qt/QML based applications as dynamic skills in Mycroft Core.

Mycroft is free and open source digital assistant written in python, Mycroft skills are the voice apps and extensions for Mycroft also written in python they can provide different functionality for users depending on what the user wants to achieve, like for example mycroft skills can be written to provide the user with weather information which can then inform the user of the current weather and display a weather forecast.

Libqmycroft aims to bring the same Mycroft extension capability to traditional input desktop / mobile applications that want to add support for a voice interface without having each application to go through writing their own web-socket / dbus interface to establish a communication layer with Mycroft.

What it does currently

Libqmycroft allows QML based applications to register themselves as dynamic skills using the Libqmycroft Mock Skills Interface and Libqmycroft SkillManager declarative API. It tries to eliminate the need to create specific python skills for traditional applications to gain voice control over the application user interface.

What it does not do currently

It does not currently fully expose or support all of Mycroft’s internal Skill Interface, so dynamic skills written are only limited to usage of only Mycroft’s Padatious intent parser framework. It cannot yet be fully used to create conversation style dynamics skills but only simple command, control, and action based reaction skills for applications wanting to implement a simple voice interface for controlling their current QML based graphical user interface.

How does it work

Everything over the socket connection

Libqmycroft establishes a web-socket connection instance for each application using its controller interface with the Mycroft message-bus, this allows the application and mycroft to exchange specially formatted messages over the local socket network which is also visible to the Mycroft skills interface.

The controller is an minified abstraction of the original Mycroft Controller class from the Mycroft GUI project, It is a singleton instance that manages web-socket connections and message exchange to the Mycroft message-bus.

Mycroft skill interface side of solution

Libqmycroft uses a middle layer called the Libqmycroft Mock Skill Interface that wraps around the original python Mycroft Skills interface class like a normal python based Mycroft Skill exposing the intent and vocab registration methods to incoming dynamic skills creation request over the Mycroft message-bus. Each incoming dynamic skills creation request over this channel registers the list of intents with their respective vocab intent file in the Padatious intent service using a combination of application provided namespace and intent file names to correctly map intents back to the registering application. At the time of registering intents the mock skill interface dynamically generates handler/callback functions that are then registered as handlers for those intent matches which are then only activated when mycroft matches the voice intent back to them in the mock skill environment, sending a call for action to the registered application over the connected instance.

Qt/QML side of solution

The SkillManager declarative API exposes a SkillManager object to QML that is responsible for managing and registering a list of SkillEntry objects, SkillEntry objects are individually a container for representing a single intent with the intent description, vocabulary file path containing words to activate the intent along with an action and to speak dialog which the later two being reactive to intent call for action. The SkillManager API also exposes functions for creating the dynamic skill and informing of intent responses when intent call for action have been invoked by voice interaction.

Work In Progress

Libqmycroft is relatively a new concept and still a much work in progress library with currently only very basic voice interface functionality of mycroft exposed to traditional input applications, lots of the actual Mycroft Skill interface is still left to be fully utilised and usable, In the ongoing development cycle the plan is to also make Adapt another lightweight intent parser available for use with intent registration coming from Libqmycroft, creating a more stable API, along with adding support for Libqmycroft to connect to Hivemind a mesh networking solution for Mycroft Instances hosted over the network.

Contributions, ideas and feedback is always welcomed from the community ! One can start contributing or experimenting with adding a simple voice interface for their QML based applications today using the library and Mycroft, documentation with complete requirements, examples and API explanation along with library source and installer scripts is available at Libqmycroft Repository Link

Categories:

Leave a Reply

Your email address will not be published. Required fields are marked *