Swiftide enables you to build indexing pipelines in a modular fashion, allowing for experimentation and blazing fast, production ready performance for Retrieval Augmented Generation (RAG).
In a nutshell, an indexing pipeline:
loads data with a loader
a loader emits nodes
nodes are transformed, enriched, chunked and embedded
… and finally Nodes are persisted into storage
In this article, we will cover the basics of indexing and querying code with Swiftide. Step by step we will set up the required dependencies and build up a pipeline. We will then manually build a query function to answer a user’s question.
To learn more about Swiftide, head over to swiftide.rs or check us out on github
Setting up the project
Before we can get started we need to set up a new Rust project and install Redis and Qdrant.
Make sure you have the Rust toolchain installed, rustup is recommended. For the example we will use OpenAI to enrich, embed and query our data.
Then install both Redis and Qdrant, either via your favourite package manager or run them as one-off via Docker.
On github you can also find a docker compose file.
Then make sure the following environment variables are configured:
Creating the Rust project
First, let’s set up the Rust project:
Next we’ll add some useful dependencies to make our life easier, and Tokio:
And add Swiftide itself:
Setting up our main function
Then set up a basic main for the project:
Give it a test:
And we’re ready to go!
Building the indexing pipeline
Any RAG pipeline is only as good as the data it ingests. For brevity, let’s assume the project is written in a single language and has some markdown like a readme. In bigger projects we might want to index documentation, websites and other code as well. That’s all possible with Swiftide, but let’s keep that out of scope for now.
Getting user input
First, let’s get some user input when we run the program. We want to know the language, the directory it should index and query, and later the query itself. Let’s use clap for this, which we added earlier.
And add the argument parsing:
Let’s run it to make sure it works:
Enabling logging
Before we continue, let’s enable logging so we can easily see if something goes wrong.
And initialize the stdout subscriber right below main:
By setting RUST_LOG to debug or info it should now be a lot easier to identify problems.
Indexing markdown files
First we will index any markdown files, like a README. There might be valuable data about what the project does and we want to be sure to have that included.
And call it from main:
This loads all markdown files into the pipeline and runs the pipeline, but it does not do anything yet.
Chunking the data
In RAG, we generally store smaller chunks than the full file. This gives more granular and detailed results and gives the opportunity to very specifically enrich. Chunking is an exercise by itself. The goal is to get small, meaningful blocks while still allowing for performance at scale.
For this we will use the markdown chunker, which under the hood uses the excellent Rust text-splitter crate. It tries to get meaningful blocks of text by using the markdown formatting as a guide.
Enriching the data with questions and answers
A common technique for RAG is to generate questions per chunk. When searching the index, this gives more overlap with queries.
Let’s use a transformer with defaults to add it. This also sets up openai as the llm. OpenAI is cheap to clone, it wraps the client with an internal Arc.
When you try to run the pipeline, it will fail with an error that no storage is set. A pipeline must always end in at least one storage.
Embedding and storing the data
Next we will embed the chunks and store them into Qdrant. The Embed transformer only supports batch mode. Depending on the LLM, chunk sizes and resources available; this is a potential parameter to tune.
Great! Let’s run this on a repository and check if it works:
Indexing code
For indexing code we will do the exact same process in a new pipeline.
Let’s give it a whirl:
1 minute and 23 seconds. Not bad!
Getting swifty
Let’s see if we can speed it up. Since requests to openai take time, we can increase the concurrency (default is number of cpus) up to whatever the rate limit allows.
There are also approximately 500 nodes to process while we batch in 100, which means we can experiment with smaller batches to get higher parallelism.
Finally, we could also split a single stream into markdown and code. Let’s do all at once and see where we’re at:
And let’s give it another whirl:
Aww yeah, 478 nodes processed in 17 seconds! For the record, splitting gets it down to ~50 seconds, and bumping the currency gets it down close to 20. The smaller embedding batches shave off the rest.
With that sorted, let’s quickly add a node cache right before splitting, so that subsequent queries will only re-index changed nodes:
Wrapping it up with a query and a poem
Swiftide does not include a querying pipeline yet. This is highly desired and in progress. At this moment in time we will do it manually
Let’s add a query to our freshly indexed data and see if we can get some answers on all this confusing code. We’ll use gpt-4o for the inference and rewrite the user query with multiple questions. In real environments you might do a lot more, but let’s leave that to an article of it’s own.
And let’s give it a final go:
What is Swiftide, you ask my muse?
A tale to tell with verses true.
A library in Rust that's spun,
For data weaving swiftly done.
With streams async, it blazes fast,
Transforms and splits, embeds and lasts.
In RAG it holds a guiding light,
For LLMs to work just right.
Born from need, where Python failed,
Swiftide's speed and grace prevailed.
Data's journey through its veins,
From chunks to nodes, it entertains.
A bosun's part in code's grand dance,
For autonomous advance.
Yet know, Swiftide's a work in flow,
With changes swift and bugs to show.
Hand in hand with Treesitter’s might,
Or OpenAI’s insight.
Qdrant and Redis join the ride,
FastEmbed stands by its side.
A project growing, feedback’s gold,
In issues, stories shall unfold.
On [swiftide.rs][1] explore,
Or GitHub pages, there’s much more.
So, here's Swiftide, a name to note,
In Rust, it sails, a data boat.