Coalescing Concurrent Work with Singleflight

Concurrent callers sometimes need the same expensive result at the same time: a short-lived credential that several requests must refresh, a remote directory that several operations need to exist, or a report that several queries want built. Running the work once per caller multiplies load on whatever produces the result, and every caller waits for its own duplicate execution.

Asyncband's singleflight::Group coalesces these calls: while work for a key is in flight, later callers for that key wait and, on success, receive a clone of the value. This post covers sharing one execution, what happens when the shared execution fails or is cancelled, and how to choose between a group and the once family, whose members retain completed state.

The Asyncband APIs below are from version 0.7.3, with the singleflight features enabled.

Share one execution per key

A group coalesces by key. work takes a key and an async function; while the function for that key is running, later calls with the same key wait for its result instead of starting their own:

use asyncband::singleflight::Group;

struct Credentials {
    refreshes: Group<&'static str, String>,
}

impl Credentials {
    async fn refresh_token(&self) -> String {
        self.refreshes
            .work("token", || async {
                // one round trip to the issuer, no matter how many callers overlap
                issue_token().await
            })
            .await
    }
}

Three callers that overlap run issue_token once. The first executes the function; the other two wait and each receive a clone of its value, so the value type must be Clone. Share an Arc<T> when the value is large or cloned often.

Keys are independent: calls with different keys never share an execution. A group does not limit overall concurrency; it collapses duplicate executions for the same key.

No cache after completion

When the function completes successfully, the group removes the key. A call that starts after the previous one finished executes its own function again. For the token sample this means that calls which do not overlap each issue a token. A real refresh path caches the token until near expiry and enters the group only on a cache miss; the group itself holds no value over for later calls.

When later callers should reuse a completed result, either compose the group with a cache of your own or use the once family below. The two production users at the end of this post both compose: the group collapses overlapping calls, and a separate cache retains the result for later callers.

Errors, cancellation, and retries

work takes an infallible function. Use try_work when the function can fail: it takes a function returning Result<V, E> and returns Result<V, E> to the caller.

An error is returned only to the caller that ran the function, not shared with callers that joined. After an error, or if the active call is cancelled or panics, the next waiting caller takes over and runs its own function while any remaining callers keep waiting. The same key can therefore execute more than once while attempts fail.

The function must not call work or try_work for its own key: it would wait for its own result. Calls for other keys proceed independently.

forget detaches a key's in-flight execution. Calls that already joined it keep their result; later callers start a fresh execution even while the old one is still running. Use it when an in-flight call is taking too long and newer callers should not queue behind it.

Choosing between a group and the Once family

Asyncband's Once family retains completed state; a group does not. The members differ in what they retain:

Asyncband API	After completion	Use
singleflight::Group	The key is removed; a later call executes again.	Overlapping calls for a key share one execution.
OnceMap	The value is retained until remove or discard.	Compute once per key and cache the result.
OnceCell	The value is retained.	Store one value from an initializer supplied at access time.
LazyCell	The value is retained.	Initialize one value with a stored function; the in-flight future survives caller cancellation.
Once	No value is stored.	Run a side effect once, without storing a value.

OnceMap is the retained counterpart of a group. Its compute runs the function once per key and serves every later caller a clone of the stored value, until remove or discard drops the entry, after which the next call computes again. try_compute takes a Result-returning function, mirroring try_work on a group. Every successful lookup returns an owned clone of the value, so prefer Arc<T> entries when values are large or frequently shared.

Failed, cancelled, or panicked attempts leave OnceCell and OnceMap entries available for another attempt. Once is likewise infallible, and a cancelled or panicked run is abandoned the same way: a later call runs again. LazyCell instead preserves the same in-flight future across caller cancellation, and a panic poisons the cell rather than starting another attempt.

In production

Reqsign signs Azure Storage requests with user delegation keys. Because the keys expire, a burst of signing requests can simultaneously find the cached key missing. Reqsign keeps one group for these refreshes: on a cache miss, callers enter try_work with a key covering the cache entry and its validity window, so overlapping misses issue one request to Azure. The closure re-checks the cache before sending the request in case another call populated a usable key after the initial miss, and a failed request returns its error to the executing caller while letting a joined caller retry. However, Reqsign writes the fetched key to the cache after try_work returns: a caller arriving after the flight finishes but before that write lands misses both the flight and the cache, issuing a second request unless the write completes before its closure's re-check runs.

OpenDAL's Google Drive backend creates directories on demand, where concurrent operations on the same path would otherwise each query or create the same directory. The backend keys a group by the parent directory's ID and the new directory's name, re-checking the path index inside try_work before querying or creating the directory. Unlike Reqsign's after-return write, OpenDAL inserts the resolved ID into the path index before the shared execution returns: publishing inside the flight ensures that on success there is never a gap where the flight is gone and the index is still empty.

For work that overlapping callers can share, a group collapses the duplicates and retains nothing. Combine it with a cache when later callers should reuse the result, or choose the once family when retention is the primary need.