Dagster
Dagster is a Python framework for data orchestration built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability.
The dagster-solvio library lets you integrate Solvio’s vector database with Dagster, making it easy to build AI-driven data pipelines. You can run vector searches and manage data directly within Dagster.
Installation
pip install dagster dagster-solvio
Example
from dagster_solvio import SolvioConfig, SolvioResource
import dagster as dg
@dg.asset
def my_table(solvio_resource: SolvioResource):
with solvio_resource.get_client() as solvio:
solvio.add(
collection_name="test_collection",
documents=[
"This is a document about oranges",
"This is a document about pineapples",
"This is a document about strawberries",
"This is a document about cucumbers",
],
)
results = solvio.query(
collection_name="test_collection", query_text="hawaii", limit=3
)
defs = dg.Definitions(
assets=[my_table],
resources={
"solvio_resource": SolvioResource(
config=SolvioConfig(
host="xyz-example.eu-central.aws.cloud.solvio.io",
api_key="<your-api-key>",
)
)
},
)
Next steps
Dagster documentation
Dagster examples