Skip to content

Job Interface

Simple job rest API

1. Submit a sqlflow job

Call this API by sending the SQL files and get the result includes the data lineage. SQLFlow job supports both of multiple files and zip archive file.

1
/gspLive_backend/sqlflow/job/submitUserJob

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/submitUserJob" -H "accept:application/json;charset=utf-8" -H "Content-Type:multipart/form-data" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE" -F "sqlfiles=@FIRST FILE PATH" -F "sqlfiles=@SECOND FILE PATH" -F "dbvendor=dbvmssql" -F "jobName=job1"

/submitUserJob

Note:

  • -H "Content-Type:multipart/form-data" is required
  • Add @ before the file path

Sample response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
{
    "code":200,
    "data":{
        "jobId":"c359aef4bd9641d697732422debd8055",
        "jobName":"job1",
        "userId":"google-oauth2|104002923119102769706",
        "dbVendor":"dbvmssql",
        "dataSource":{

        },
        "fileNames":["1.sql","1.zip"],
        "createTime":"2020-12-15 15:14:39",
        "status":"create"
    }
}

Please records the jobId field.

2. Get job status

1
/gspLive_backend/sqlflow/job/displayUserJobSummary

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/displayUserJobSummary" -F "jobId=c359aef4bd9641d697732422debd8055" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE"

/displayUserJobSummary

Sample response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
{
  "code":200,
  "data":{
    "jobId":"c359aef4bd9641d697732422debd8055",
    "jobName":"job1",
    "userId":"google-oauth2|104002923119102769706",
    "dbVendor":"dbvmssql",
    "dataSource":{

    },
    "fileNames":["1.sql","1.zip"],
    "createTime":"2020-12-15 15:14:39",
    "status":"success",
    "sessionId":"fe5898d4e1b1a7782352b50a8203ca24c04f5513446e9fb059fc4d584fab4dbf_1608045280033"
  }
}

1
/gspLive_backend/sqlflow/job/displayUserJobsSummary

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/displayUserJobsSummary" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE"

/displayUserJobsSummary

Sample Response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
{
  "code": 200,
  "data": {
    "total": 2,
    "success": 2,
    "partialSuccess": 0,
    "fail": 0,
    "jobIds": [
      "939fdbbaf52b45139b86c0761d6036b0",
      "44bf162fc5e04ab68289e6e784875203"
    ],
    "jobDetails": [
      {
        "jobId": "939fdbbaf52b45139b86c0761d6036b0",
        "jobName": "yuanhao1",
        "userId": "auth0|632xxxx",
        "userEmail": "xxx@xxx.com",
        "dbVendor": "dbvoracle",
        "showRelationType": "fdd",
        "ignoreRecordSet": true,
        "dataflowOfAggregateFunction": "direct",
        "showConstantTable": false,
        "showTransform": false,
        "ignoreFunction": true,
        "fileNames": [
          "test.sql"
        ],
        "createTime": "2022-10-04 10:45:04",
        "status": "success",
        "sessionId": "24a4455c71fa35c0393d5747f9c23a9d99f32fa4c130a6b8da8d6a7db8d157ae_1664880305725",
        "executeTime": "0min 1sec",
        "executeTimeMillis": 0,
        "onTop": false,
        "parallel": false,
        "persist": false,
        "incremental": false,
        "schedulable": false,
        "cancelSchedule": false,
        "lastExecuteTime": "2022-10-04 10:45:05",
        "subJobs": [],
        "version": "5.2.0"
      },
      {
        "jobId": "44bf162fc5e04ab68289e6e784875203",
        "jobName": "sqlserver sample lineage",
        "userId": "auth0|6326xxxx",
        "userEmail": "xxxx@xxxx.com",
        "dbVendor": "dbvmssql",
        "showRelationType": "fdd",
        "ignoreRecordSet": true,
        "showConstantTable": false,
        "showTransform": false,
        "ignoreFunction": true,
        "fileNames": [
          "metadata.zip"
        ],
        "createTime": "2022-09-18 06:46:20",
        "status": "success",
        "sessionId": "e1bc494a6f683c9e724c5f018d2a0c96d80304a3ddb025caf87dc1ac19e1a785_1636819107871",
        "hasMetadata": true,
        "executeTime": "0min 10sec",
        "executeTimeMillis": 0,
        "onTop": false,
        "parallel": false,
        "persist": false,
        "incremental": false,
        "schedulable": false,
        "cancelSchedule": true
      }
    ]
  }
}

3. Export data lineage

When the job status is success, you can export the data lineage in json, csv, graphml formats

1
/gspLive_backend/sqlflow/job/exportLineageAsJson

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/exportLineageAsJson" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE" -F "jobId=c359aef4bd9641d697732422debd8055" --output lineage.json

/exportLineageAsJson

Note:

If you want to get table to table relation, please add option -F "tableToTable=true"

Sample Response is a file in Json format:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
{
    "jobId":"939fdbbaf52b45139b86c0761d6036b0",
    "code":200,
    "data":{
        "mode":"global",
        "summary":{
            "schema":0,
            "process":1,
            "database":0,
            "view":0,
            "mostRelationTables":[],
            "column":3,
            "relationship":0,
            "table":1
        },
        "sqlflow":{
            "dbvendor":"dbvoracle",
            "relationships":[],
            "dbobjs":[
                {
                    "queryHashId":"04ebc5aec1a07e1db80b0bc798742875",
                    "name":"QUERY INSERT-1",
                    "coordinates":[
                        {
                            "hashCode":"0",
                            "x":1,
                            "y":1
                        },
                        {
                            "hashCode":"0",
                            "x":1,
                            "y":73
                        }
                    ],
                    "id":"8",
                    "type":"process"
                },
                {
                    "columns":[
                        {
                            "name":"ID",
                            "coordinates":[
                                {
                                    "hashCode":"0",
                                    "x":1,
                                    "y":28
                                },
                                {
                                    "hashCode":"0",
                                    "x":1,
                                    "y":30
                                }
                            ],
                            "id":"5"
                        },
                        {
                            "name":"FIRST_NAME",
                            "coordinates":[
                                {
                                    "hashCode":"0",
                                    "x":1,
                                    "y":32
                                },
                                {
                                    "hashCode":"0",
                                    "x":1,
                                    "y":42
                                }
                            ],
                            "id":"6"
                        },
                        {
                            "name":"LAST_NAME",
                            "coordinates":[
                                {
                                    "hashCode":"0",
                                    "x":1,
                                    "y":44
                                },
                                {
                                    "hashCode":"0",
                                    "x":1,
                                    "y":53
                                }
                            ],
                            "id":"7"
                        }
                    ],
                    "name":"RAW_CUSTOMERS",
                    "coordinates":[
                        {
                            "hashCode":"0",
                            "x":1,
                            "y":13
                        },
                        {
                            "hashCode":"0",
                            "x":1,
                            "y":26
                        }
                    ],
                    "id":"4",
                    "type":"table"
                }
            ]
        },
        "graph":{
            "relationshipIdMap":{},
            "elements":{
                "tables":[],
                "edges":[]
            },
            "tooltip":{},
            "listIdMap":{}
        }
    },
    "sessionId":"24a4455c71fa35c0393d5747f9c23a9d99f32fa4c130a6b8da8d6a7db8d157ae_1664880305725"
}

1
/gspLive_backend/sqlflow/job/exportFullLineageAsCsv

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/exportFullLineageAsCsv" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE" -F "jobId=c359aef4bd9641d697732422debd8055" --output lineage.csv

/exportFullLineageAsCsv

Note:

If you want to get table to table relation, please add option -F "tableToTable=true"

If you want to change csv delimiter, please add option -F "delimiter=\"

1
/gspLive_backend/sqlflow/job/exportLineageAsGraphml

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/exportLineageAsGraphml" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE" -F "jobId=c359aef4bd9641d697732422debd8055" --output lineage.graphml

/exportLineageAsGraphml

Note:

If you want to get table to table relation, please add option -F "tableToTable=true"

Regular job rest API

1. Submit a regular job

Call this API by sending the SQL files and get the result includes the data lineage. SQLFlow job supports both of multiple files and zip archive file.

Set incremental=true If the job is incremental.

  • jobId should be null for the first submit and please note down the jobId field from response message
  • jobId cannot be null for next submit. Give the jobId which is returned in the first submit response.
1
/gspLive_backend/sqlflow/job/submitPersistJob

/submitPersistJob

Example in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/submitPersistJob" -H "accept:application/json;charset=utf-8" -H "Content-Type:multipart/form-data" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE" -F "sqlfiles=@FIRST FILE PATH" -F "sqlfiles=@SECOND FILE PATH" -F "dbvendor=dbvmssql" -F "jobName=job1" -F "incremental=true"

Incremental submit in Curl

1
curl -X POST "https://api.gudusoft.com/gspLive_backend/sqlflow/job/submitPersistJob" -H "accept:application/json;charset=utf-8" -H "Content-Type:multipart/form-data" -F "userId=YOUR USER ID HERE" -F "token=YOUR TOKEN HERE" -F "sqlfiles=@FIRST FILE PATH" -F "sqlfiles=@SECOND FILE PATH" -F "dbvendor=dbvmssql" -F "jobName=job1" -F "incremental=true" -F "jobId=JobId OF FIRST SUBMIT"

Note:

  • -H "Content-Type:multipart/form-data" is required
  • Add @ before the file path

Return data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
{
    "code":200,
    "data":{
        "jobId":"c359aef4bd9641d697732422debd8055",
        "jobName":"job1",
        "userId":"google-oauth2|104002923119102769706",
        "dbVendor":"dbvmssql",
        "dataSource":{

        },
        "fileNames":["1.sql","1.zip"],
        "createTime":"2020-12-15 15:14:39",
        "status":"create"
    }
}

Please records the jobId field for the further usage.