๐Ÿ“‹ Overview ๐ŸŽฎ Demo ๐Ÿ”Œ API ๐Ÿ’ก Examples ๐Ÿ“š Resources

๐Ÿ”Œ WIA-DATA-010 API Reference

Base URL

https://api.wia-data-integration.io/v1

All API endpoints are relative to this base URL. Authentication is required for all requests.

Authentication

Use Bearer token authentication with your API key:

Authorization: Bearer YOUR_API_KEY

Pipeline Management

POST /pipelines

Create a new data integration pipeline

Request Body

Parameter Type Required Description
name string Yes Pipeline name
source object Yes Source configuration
destination object Yes Destination configuration
transformations array No List of transformation rules
schedule string No Cron expression for scheduling

Example Request

{
  "name": "customer-data-sync",
  "source": {
    "type": "postgresql",
    "host": "db.example.com",
    "port": 5432,
    "database": "production",
    "schema": "public",
    "table": "customers"
  },
  "destination": {
    "type": "snowflake",
    "account": "xyz12345",
    "warehouse": "COMPUTE_WH",
    "database": "ANALYTICS",
    "schema": "STAGING",
    "table": "customers"
  },
  "transformations": [
    {
      "type": "rename",
      "from": "cust_id",
      "to": "customer_id"
    },
    {
      "type": "filter",
      "condition": "status = 'active'"
    }
  ],
  "schedule": "0 */6 * * *"
}

Response

{
  "pipeline_id": "pip_a1b2c3d4e5f6",
  "name": "customer-data-sync",
  "status": "created",
  "created_at": "2025-01-15T10:30:00Z"
}

GET /pipelines/{pipeline_id}

Get pipeline details and status

Response

{
  "pipeline_id": "pip_a1b2c3d4e5f6",
  "name": "customer-data-sync",
  "status": "running",
  "last_run": {
    "started_at": "2025-01-15T12:00:00Z",
    "completed_at": "2025-01-15T12:15:30Z",
    "records_processed": 125000,
    "status": "success"
  },
  "next_run": "2025-01-15T18:00:00Z"
}

POST /pipelines/{pipeline_id}/run

Manually trigger a pipeline run

Response

{
  "run_id": "run_x9y8z7w6v5u4",
  "pipeline_id": "pip_a1b2c3d4e5f6",
  "status": "running",
  "started_at": "2025-01-15T14:30:00Z"
}

DELETE /pipelines/{pipeline_id}

Delete a pipeline

Response

{
  "message": "Pipeline deleted successfully",
  "pipeline_id": "pip_a1b2c3d4e5f6"
}

Connectors

GET /connectors

List all available connectors

Response

{
  "connectors": [
    {
      "id": "postgresql",
      "name": "PostgreSQL",
      "type": "source",
      "category": "database",
      "supported_versions": ["12", "13", "14", "15"]
    },
    {
      "id": "snowflake",
      "name": "Snowflake",
      "type": "destination",
      "category": "warehouse"
    },
    {
      "id": "kafka",
      "name": "Apache Kafka",
      "type": "both",
      "category": "streaming"
    }
  ]
}

Monitoring & Logs

GET /pipelines/{pipeline_id}/runs

Get run history for a pipeline

Query Parameters

Parameter Type Description
limit integer Number of runs to return (default: 20)
status string Filter by status: success, failed, running

GET /runs/{run_id}/logs

Get detailed logs for a specific run

Response

{
  "run_id": "run_x9y8z7w6v5u4",
  "logs": [
    {
      "timestamp": "2025-01-15T14:30:00Z",
      "level": "INFO",
      "message": "Pipeline started"
    },
    {
      "timestamp": "2025-01-15T14:30:05Z",
      "level": "INFO",
      "message": "Connected to source: PostgreSQL"
    },
    {
      "timestamp": "2025-01-15T14:35:20Z",
      "level": "SUCCESS",
      "message": "Processed 125000 records"
    }
  ]
}

Error Codes

Code Status Description
AUTH_001 401 Invalid or missing API key
PIPE_001 404 Pipeline not found
PIPE_002 400 Invalid pipeline configuration
CONN_001 500 Connection to source failed
CONN_002 500 Connection to destination failed
RATE_001 429 Rate limit exceeded

Rate Limits

API requests are subject to the following rate limits:

Rate limit headers are included in every response:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 95
X-RateLimit-Reset: 1610723400