Module work

Zephyr Work Queues

Zephyr Work Queues and Work

Zephyr has a mechanism called a Workqueue.

Each workqueue is backed by a single Zephyr thread, and has its own stack. The work queue consists of a FIFO queue of work items that will be run consecutively on that thread. The underlying types are k_work_q for the work queue itself, and k_work for the worker.

In addition to the simple schedulable work, Zephyr also has two additional types of work: k_work_delayable which can be scheduled to run in the future, and k_work_poll, described as triggered work in the docs. This can be scheduled to run when various items within Zephyr become available. This triggered work also has a timeout. In this sense the triggered work is a superset of the other types of work. Both delayable and triggered work are implemented by having the k_work embedded in their structure, and Zephyr schedules the work when the given reason happens.

Zephyr’s work queues can be used in different ways:

• Work can be scheduled as needed. For example, an IRQ handler can queue a work item to process data it has received from a device.
• Work can be scheduled periodically.

As most C use of Zephyr statically allocates things like work, these are typically rescheduled when the work is complete. The work queue scheduling functions are designed, and intended, for a given work item to be able to reschedule itself, and such usage is common.

Waitable events

The triggerable work items can be triggered to wake on a set of any of the following:

• A signal. k_poll_signal is a type used just for waking work items. This works similar to a binary semaphore, but is lighter weight for use just by this mechanism.
• A semaphore. Work can be scheduled to run when a k_sem is available. Since sys::sync::Semaphore is built on top of k_sem, the “take” operation for these semaphores can be a trigger source.
• A queue/FIFO/LIFO. The queue is used to implement sync::channel and thus any blocking operation on queues can be a trigger source.
• Message Queues, and Pipes. Although not yet provided in Rust, these can also be a source of triggering.

It is important to note that the trigger source may not necessarily still be available by the time the work item is actually run. This depends on the design of the system. If there is only a single waiter, then it will still be available (the mechanism does not have false triggers, like CondVar).

Also, note, specifically, that Zephyr Mutexes cannot be used as a trigger source. That means that locking a sync::Mutex shouldn’t be use within work items. There is another kio::sync::Mutex, which is a simplified Mutex that is implemented with a Semaphore that can be used from work-queue based code.

Rust Future

The rust language, also has built-in support for something rather similar to Zephyr work queues. The main user-visible type behind this is Future. The rust compiler has support for functions, as well as code blocks to be declared as async. For this code, instead of directly returning the given data, returns a Future that has as its output type the data. What this does is essentially capture what would be stored on the stack to maintain the state of that code into the data of the Future itself. For rust code running on a typical OS, a crate such as Tokio provides what is known as an executor, which implements the schedule for these Futures as well as provides equivalent primitives for Mutex, Semaphores and channels for this code to use for synchronization.

It is notable that the Zephyr implementation of Future operations under a fairly simple assumption of how this scheduling will work. Each future is invoked with a Context, which contains a dynamic Waker that can be invoked to schedule this Future to run again. This means that the primitives are typically implemented above OS primitives, where each manages wake queues to determine the work that needs to be woken.

Bringing it together.

There are a couple of issues that need to be addressed to bring work-queue support to Rust. First is the question of how they will be used. On the one hand, there are users that will definitely want to make use of async in rust, and it is important to implement a executor, similar to Tokio, that will schedule this async code. On the other hand, it will likely be common for others to want to make more direct use of the work queues themselves. As such, these users will want more direct access to scheduling and triggering of work.

Future erasure

One challenge with using Future for work is that the Future type intentionally erases the details of scheduling work, reducing it down to a single Waker, which similar to a trait, has a wake method to cause the executor to schedule this work. Unfortunately, this simple mechanism makes it challenging to take advantage of Zephyr’s existing mechanisms to be able to automatically trigger work based on primitives.

As such, what we do is have a structure Work that contains both a k_work_poll as well as Context from Rust. Our handler can use a mechanism similar to C’s CONTAINER_OF macro to recover this outer structure.

There is some extra complexity to this process, as the Future we are storing associated with the work is ?Sized, since each particular Future will have a different size. As such, it is not possible to recover the full work type. To work around this, we have a Sized struct at the beginning of this structure, that along with judicious use of #[repr(C)] allows us to recover this fixed data. This structure contains the information needed to re-schedule the work, based on what is needed.

Ownership

The remaining challenge with implementing k_work for Rust is that of ownership. The model taken here is that the work items are held in a Box that is effectively owned by the work itself. When the work item is scheduled to Zephyr, ownership of that box is effectively handed off to C, and then when the work item is called, the Box re-constructed. This repeats until the work is no longer needed (e.g. when a Future::poll returns Ready), at which point the work will be dropped.

There are two common ways the lifecycle of work can be managed in an embedded system:

• A set of Future’s are allocated once at the start, and these never return a value. Work Futures inside of this (which correspond to .await in async code) can have lives and return values, but the main loops will not return values, or be dropped. Embedded Futures will typically not be boxed.
• Work will be dynamically created based on system need, with threads using kio::spawn to create additional work (or creating the Work items directly). These can use join or join_async to wait for the results.

One consequence of the ownership being passed through to C code is that if the work cancellation mechanism is used on a work queue, the work items themselves will be leaked.

The Future mechanism in Rust relies on the use of Pin to ensure that work items are not moved. We have the same requirements here, although currently, the pin is only applied while the future is run, and we do not expose the Box that we use, thus preventing moves of the work items.

The work queues themselves

Workqueues themselves are built using WorkQueueBuilder. This needs a statically defined stack. Typical usage will be along the lines of:

kobj_define! {
  WORKER_STACK: ThreadStack<2048>;
}
// ...
let main_worker = Box::new(
    WorkQueueBuilder::new()
        .set_priority(2).
        .set_name(c"mainloop")
        .set_no_yield(true)
        .start(MAIN_LOOP_STACK.init_once(()).unwrap())
    );

let _ = zephyr::kio::spawn(
    mainloop(), // Async or function returning Future.
    &main_worker,
    c"w:mainloop",
);

...

// Leak the Box so that the worker is never freed.
let _ = Box::leak(main_worker);

It is important that WorkQueues never be dropped. It has a Drop implementation that invokes panic. Zephyr provides no mechanism to stop work queue threads, so dropping would result in undefined behavior.

Current Status

Although Zephyr has 3 types of work queues, the k_work_poll is sufficient to implement all of the behavior, and this implementation only implements this type. Non Future work could be built around the other work types.

As such, this means that manually constructed work is still built using Future. The _async primitives throughout this crate can be used just as readily by hand-written Futures as by async code. Notable, the use of Signal will likely be common, along with possible timeouts.

Modules

• futures
  Zephyr work wrappers targeted for the Future type.

Structs

• Signal
  A Rust wrapper for k_poll_signal.
• Work
  A basic Zephyr work item.
• WorkQueue
  A running work queue thread.
• WorkQueueBuilder
  A builder for work queues themselves.

Enums

• SubmitResult
  Possible returns from work queue submission.

Traits

• SimpleAction
  A simple action that just does something with its data.

Functions

• get_current_workq
  Retrieve the current work queue, if we are running within one.