Data & ingestion

This page walks the complete pipeline workflow with real request/response shapes. All calls require authentication (examples use a cookie jar).

1. Create a group

Data sources belong to a group, so you need one first:

curl -b cookies.txt -X POST $BASE/user_groups \
  -H 'Content-Type: application/json' \
  -d '{"group_name": "operations", "group_permission": "write"}'

{ "created_group_id": "7f2c8878-85af-4e18-bc60-b80852f6a6b6" }

group_name: 3–50 chars, letters/numbers/underscores. group_permission: read, write, or none.

Other group operations: GET /user_groups (list), PUT /user_groups/{id}/members/{email} (add member), DELETE /user_groups/{id}/members/{email} (remove), PUT /user_groups/{id}/parent with {"parent_group_id": "…"} (build a hierarchy; cycles are rejected), GET /user_groups/tree (full tree).

2. Create an SMB data source

curl -b cookies.txt -X POST $BASE/data_sources/smb/<group_id> \
  -H 'Content-Type: application/json' \
  -d '{
    "data_source": { "data_source_name": "Company shared drive" },
    "smb_source": {
      "remote_system_username": "svc_insight",
      "remote_smb_password": "…",
      "remote_system_name": "FILESERVER01",
      "local_system_name": "insight_ai",
      "default_ip": "192.168.1.100",
      "port": 445
    }
  }'

The server verifies the credentials by connecting to the share before saving; on failure you get 400 {"detail": "Invalid SMB credentials"} and nothing is created. On success the response contains the new data_source.id you’ll use next.

Browse what the source can see (recursive first level of every share):

curl -b cookies.txt $BASE/data_sources/smb/list/<data_source_id>
# → {"shares": {"finance-share": ["Reports", "Policies"], …}}

Shares the account can’t open come back as {"error": …} entries; an unreachable server yields 503.

3. Create a collection

curl -b cookies.txt -X POST $BASE/collections \
  -H 'Content-Type: application/json' \
  -d '{"name": "company_docs"}'

Names allow letters, numbers, and underscores, and must be unique. GET /collections lists them; DELETE /collections/{id} removes one and all its indexed content.

4. Create a pipeline

curl -b cookies.txt -X POST $BASE/data_pipelines \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "Shared drive → company_docs",
    "description": "Indexes the company shared drive.",
    "data_source_id": "<data_source_id>",
    "collection_id": "<collection_id>"
  }'

The response includes the pipeline id and its status (initially inactive). One pipeline per (source, collection) pair — duplicates return 400.

5. Trigger ingestion and poll

curl -b cookies.txt -X POST $BASE/data_pipelines/<pipeline_id>/ingest

Ingestion is asynchronous — the call returns immediately with a job record, and the work runs in a background queue (one job at a time across the whole instance). Poll for progress:

curl -b cookies.txt $BASE/data_pipelines/<pipeline_id>/ingest/status

{
  "job_id": "2a93e5c6-95dd-42e4-8ad0-1c32f5cfb643",
  "pipeline_id": "36bb68fb-7753-42ec-a205-e307942efa17",
  "status": "completed",
  "entities_discovered": 8,
  "entities_processed": 8,
  "current_entity": "Processing: /Policies/vacation_policy.md",
  "error_message": null,
  "progress_percentage": 100.0,
  "started_at": "2026-07-19T09:12:02.258733",
  "completed_at": "2026-07-19T09:12:14.485127"
}

Job status moves processing → completed or failed (with error_message set). Triggering while a job is already running returns 409 {"detail": "Ingestion already in progress"}.

A robust wait loop:

while true; do
  S=$(curl -s -b cookies.txt $BASE/data_pipelines/$PIPELINE/ingest/status \
      | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')
  [ "$S" = "completed" ] && break
  [ "$S" = "failed" ] && { echo "ingestion failed"; exit 1; }
  sleep 5
done