Data & ingestion

This page walks the complete pipeline workflow with real request/response shapes. All calls require authentication (examples use a cookie jar).

Access control in one paragraph

Collections, and everything that touches them, are gated by owner-group membership: you can see and use a collection only if you belong to the group that owns it (direct membership — parent groups don’t inherit). Data sources are gated the same way through their owner group. Denied or nonexistent ids both return 404, deliberately — the API doesn’t reveal which ids exist. The one place you’ll see a 403 is creating a collection in a group you’re not a member of.

1. Create a group

Data sources and collections belong to a group, so you need one first:

curl -b cookies.txt -X POST $BASE/user_groups \
  -H 'Content-Type: application/json' \
  -d '{"group_name": "operations", "group_permission": "write"}'

{ "created_group_id": "7f2c8878-85af-4e18-bc60-b80852f6a6b6" }

group_name: 3–50 chars, letters/numbers/underscores/hyphens. group_permission: read, write, or none.

Other group operations: GET /user_groups (list), PUT /user_groups/{id}/members/{email} (add member), DELETE /user_groups/{id}/members/{email} (remove), PUT /user_groups/{id}/parent with {"parent_group_id": "…"} (build a hierarchy; cycles are rejected), GET /user_groups/tree (full tree).

Deleting a group that still owns data sources returns 409 with a message naming the sources — delete or migrate them first.

2. Create a data source

Creating a source is now transactional across three objects: the server verifies the connection, then auto-provisions a collection (named <slug-of-source-name>_<8 hex chars>) and a pipeline wired between them. The response carries all of it, so one call takes you from nothing to ready-to-ingest. If any part fails, the whole thing rolls back.

SMB

curl -b cookies.txt -X POST $BASE/data_sources/smb/<group_id> \
  -H 'Content-Type: application/json' \
  -d '{
    "data_source": { "data_source_name": "Company shared drive" },
    "smb_source": {
      "remote_system_username": "svc_insight",
      "remote_smb_password": "…",
      "remote_system_name": "FILESERVER01",
      "local_system_name": "insight_ai",
      "default_ip": "192.168.1.100",
      "port": 445
    }
  }'

The server verifies the credentials by connecting to the share before saving; on failure you get 400 {"detail": "Invalid SMB credentials"} and nothing is created. On success the response contains the data_source, the smb_source details, and the auto-provisioned pipeline (whose collection_id points at the new collection).

Browse what the source can see (recursive first level of every share):

curl -b cookies.txt $BASE/data_sources/smb/list/<data_source_id>
# → {"shares": {"finance-share": ["Reports", "Policies"], …}}

Shares the account can’t open come back as {"error": …} entries; an unreachable server yields 503.

Google Drive

Drive sources require the instance to have a Google OAuth client configured (see self-hosting configuration), and the calling user to have connected their Google account first:

# Is Drive connected for this user?
curl -b cookies.txt $BASE/integrations/google/drive
# → {"connected": true, "provider_subject": "…", "scopes": […], …}

# Get the consent URL (open it in a browser to grant access)
curl -b cookies.txt -X POST "$BASE/integrations/google/drive/connect?return_to=integrations"
# → {"auth_url": "https://accounts.google.com/…"}

# Disconnect
curl -b cookies.txt -X DELETE $BASE/integrations/google/drive

Then create the source (owner group goes in the body’s data_source object):

curl -b cookies.txt -X POST $BASE/data_sources/gdrive/ \
  -H 'Content-Type: application/json' \
  -d '{"data_source": {"data_source_name": "My Drive", "owner_group_id": "<group_id>"}}'

Like SMB, the response includes the auto-provisioned pipeline. What gets ingested is whatever files the grant covers:

# Register specific files (ids from the Google Picker)
curl -b cookies.txt -X POST $BASE/data_sources/gdrive/<data_source_id>/files \
  -H 'Content-Type: application/json' \
  -d '{"file_ids": ["<drive_file_id>", …]}'

# Or register everything the connected account has granted
curl -b cookies.txt -X POST $BASE/data_sources/gdrive/<data_source_id>/ingest_all

ingest_all returns 409 if the grant covers no files yet — with the drive.file scope, Google only shares files the user has explicitly picked. GET /data_sources/gdrive/picker_token mints a short-lived access token ({access_token, expires_at}) for embedding Google’s file picker in your own UI; 401 means the stored grant is dead and Drive needs reconnecting.

Delete a source

curl -b cookies.txt -X DELETE $BASE/data_sources/<data_source_id>

Requires owner-group membership. This also deletes the auto-provisioned collection and pipeline — unless another pipeline still targets that collection, in which case the collection is left alone.

3. Collections

You usually don’t create collections by hand any more — source creation provisions one. The endpoints are still there when you want a custom setup:

curl -b cookies.txt -X POST $BASE/collections \
  -H 'Content-Type: application/json' \
  -d '{"name": "company_docs", "owner_group_id": "<group_id>"}'

owner_group_id is required, and you must be a member of that group (403 "You must be a member of the owner group to create a collection in it."; 400 if the group doesn’t exist or the name is taken). Names allow letters, numbers, and underscores, and must be unique.

GET /collections lists only the collections you can access. DELETE /collections/{id} removes one and all its indexed content — 404 if it doesn’t exist or isn’t yours to touch.

4. Create a pipeline

The auto-provisioned pipeline covers the common case; create one manually when you want a different source/collection wiring:

curl -b cookies.txt -X POST $BASE/data_pipelines \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Shared drive → company_docs",
    "description": "Indexes the company shared drive.",
    "data_source_id": "<data_source_id>",
    "collection_id": "<collection_id>"
  }'

You need access to both the source and the collection (404 otherwise). The response includes the pipeline id and its status (initially inactive). One pipeline per (source, collection) pair — duplicates return 400.

5. Trigger ingestion and poll

curl -b cookies.txt -X POST $BASE/data_pipelines/<pipeline_id>/ingest

Ingestion is asynchronous — the call returns immediately with a job record, and the work runs in a background queue (one job at a time across the whole instance). Access to the pipeline’s source and collection is re-checked at trigger time. Poll for progress:

curl -b cookies.txt $BASE/data_pipelines/<pipeline_id>/ingest/status

{
  "job_id": "2a93e5c6-95dd-42e4-8ad0-1c32f5cfb643",
  "pipeline_id": "36bb68fb-7753-42ec-a205-e307942efa17",
  "status": "completed",
  "entities_discovered": 8,
  "entities_processed": 8,
  "current_entity": "Processing: /Policies/vacation_policy.md",
  "error_message": null,
  "progress_percentage": 100.0,
  "started_at": "2026-08-29T09:12:02.258733",
  "completed_at": "2026-08-29T09:12:14.485127"
}

Job status moves processing → completed or failed (with error_message set). progress_percentage is null until file discovery has counted something. Two status codes to handle deliberately:

Triggering while a job is already running returns 409 {"detail": "Ingestion already in progress for this pipeline"}.
Polling a pipeline that has never run returns 202 with {"detail": "No ingestion jobs found for this pipeline"} — an odd choice of code, but that’s what ships; don’t treat 202 as a job record.

A robust wait loop:

while true; do
  S=$(curl -s -b cookies.txt $BASE/data_pipelines/$PIPELINE/ingest/status \
      | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')
  [ "$S" = "completed" ] && break
  [ "$S" = "failed" ] && { echo "ingestion failed"; exit 1; }
  sleep 5
done