Swiftide 0.26 - Streaming agents

Published: at by Timon Vonk

Funny how time flies and you forget to write a blog post every time there is a major release. We are now at 0.26, and a lot has happened since our last update (January, 0.16!). We have been working hard on building out the agent framework, fixing bugs, and adding features. Shout out to all the contributors who have helped us along the way, and to all the users who have provided feedback and suggestions.

Swiftide is a Rust library for building LLM applications. Index, query, run agents, and bring your experiments right to production.

To get started with Swiftide, head over to swiftide.rs, check us out on github, or hit us up on discord.

Finally they stream!

Better late than never, agents can now stream their output. Under the hood, the ChatCompletion trait received an additional method called complete_stream, which returns a stream of responses, both the accumulated response and the delta. All OpenAI like providers and Anthropic are supported. We decided on including the accumulated response for convenience. Let us know if that is too many bytes for you, and we’re happy to take a look.

The kicker is that it can also be used with agents, like this:

    agents::Agent::builder()
        .llm(&anthropic)
        .on_stream(|_agent, response| {
            // We print the message chunk if it exists. Streamed responses also include
            // the full response (without tool calls) in `message` and an `id` to map them to
            // previous chunks for convenience.
            //
            // The agent uses the full assembled response at the end of the stream.
            if let Some(delta) = &response.delta {
                print!(
                    "{}",
                    delta
                        .message_chunk
                        .as_deref()
                        .map(str::to_string)
                        .unwrap_or_default()
                );
            };


            Box::pin(async move { Ok(()) })
        })
        // Every message added by the agent will be printed to stdout
        .on_new_message(move |_, msg| {
            let msg = msg.to_string();
            Box::pin(async move {
                println!("\n---\nFinal message:\n {msg}");
                Ok(())
            })
        })
        .limit(5)
        .build()?
        .query("Why is the rust programming language so good?")
        .await?;

Streaming the response back to users makes for a much snappier user experience. We’ve also implemented it in kwaak.

The completion response (streaming and non-streaming) now also includes the token usage.

MCP Support

Agents can now use tools provided by MCP (Model Context Protocol). The ecosystem is growing rapidly and there are quite a few cool tools available. Under the hood we’re using the ‘official’ Rust implementation. We don’t support creating MCP servers, as I think it’s a bit out of scope.

Here is an example of adding MCP tools to an agent:

    // First set up our client info to identify ourselves to the server
    let client_info = ClientInfo {
        client_info: Implementation {
            name: "swiftide-example".into(),
            version: env!("CARGO_PKG_VERSION").into(),
        },
        ..Default::default()
    };


    // Use `rmcp` to start the server
    let running_service = client_info
        .serve(TokioChildProcess::new(
            tokio::process::Command::new("npx")
                .args(["-y", "@modelcontextprotocol/server-everything"]),
        )?)
        .await?;


    // Create a toolbox from the running server, and only use the `add` tool
    //
    // A toolbox reveals it's tools to the swiftide agent the first time it starts (if the state of
    // the agent was pending). You can add as many toolboxes as you want. MCP services are an
    // implementation of a toolbox. A list of tools is another.
    let everything_toolbox = McpToolbox::from_running_service(running_service)
        .with_whitelist(["add"])
        .to_owned();


    agents::Agent::builder()
        .llm(&openai)
        // Add the toolbox to the agent
        .add_toolbox(everything_toolbox)
        // Every message added by the agent will be printed to stdout
        .on_new_message(move |_, msg| {
            let msg = msg.to_string();
            let tx = tx.clone();
            Box::pin(async move {
                tx.send(msg).unwrap();
                Ok(())
            })
        })
        .build()?
        .query("Use the add tool to add 1 and 2")
        .await?;

Resuming agents from history

You can now resume agents from a pre-existing history. Technically this was already possible, we’ve made it a bit easier. By creating the agent context from an existing history, the agent will resume where it left off:

  let mut first_agent = agents::Agent::builder().llm(&openai).build()?;


  first_agent.query("Say hello!").await?;


  // Let's store the messages in a database, retrieve them back, and start a new agent
  let stored_history = serde_json::to_string(&first_agent.history().await)?;
  let retrieved_history: Vec<_> = serde_json::from_str(&stored_history)?;


  let restored_context = DefaultContext::default()
      .with_message_history(retrieved_history)
      .to_owned();


  let mut second_agent = agents::Agent::builder()
      .llm(&openai)
      .context(restored_context)
      // We'll use the one from the first agent, alternatively we could also pop it from the
      // previous history and add a new one here
      .no_system_prompt()
      .build()?;


  second_agent.query("What did you say?").await?;


  Ok(())

So much more

Since January there have been so many improvements, new integrations, and a myriad of fixes, it’s hard to keep track. Luckily we have a changelog here Some more highlights since then:

Feedback, suggestions, ideas, and contributions are super welcome. Swiftide aims to make LLM app development in Rust easier, while providing opinionated building blocks to get you started. It’s your feedback that makes it worthwhile <3.

To get started with Swiftide, head over to swiftide.rs or check us out on github.