数据质量API
外部应用可以通过REST API查询数据质量数据。
查询数据质量检查规则
参数说明
| 参数名 | 描述 | 示例 |
|---|---|---|
| srcDatasource | 数据源名称 | hive_ods |
| srcTable | 数据表名称 | ods_event_track_record |
| srcField | 字段名称 | one_id |
| name | 规则名称 | 空值检查 |
| status | 状态 | create/check_enable/enable/check_disable/disable |
| current | 分页序号 | 1 |
| size | 分页行数 | 10 |
GET /data-quality/check-rule/page
{
"records": [
{
"createDeptId": "4",
"createBy": "5",
"updateBy": "5",
"createTime": "2022-07-29 15:35:56",
"updateTime": "2022-07-31 16:48:07",
"id": "1552920845936533506",
"name": "空值检查",
"description": "",
"srcConnectorType": 0,
"srcDatasourceId": 1,
"srcDatasource": "mysql_test_data",
"srcTable": "goodtbl",
"srcField": "serialNumber",
"targetConnectorType": null,
"targetDatasourceId": null,
"targetDatasource": "",
"targetTable": "",
"targetField": null,
"status": "check_disable",
"ruleId": 1,
"ruleModelDim": "完整性",
"ruleParams": {
"src_connector_type": 0,
"src_datasource_id": 0,
"src_table": "",
"src_filter": "",
"src_field": "",
"statistics_name": "",
"check_type": "2",
"operator": "3",
"threshold": "0",
"failure_strategy": "0",
"target_connector_type": 0,
"target_datasource_id": 0,
"target_table": "",
"target_filter": "",
"mapping_columns": [],
"statistics_execute_sql": "",
"comparison_name": "",
"comparison_execute_sql": "",
"comparison_type": "8",
"writer_connector_type": 0,
"writer_datasource_id": 0,
"target_field": "",
"field_length": 20,
"logic_operator": ">",
"regexp_pattern": "",
"deadline": "CURRENT_DATE()",
"datetime_format": "yyyy-MM-dd",
"enum_list": "",
"begin_time": "DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY)"
},
"weight": 5,
"version": "1.0"
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
查询数据质量检查结果
参数说明
| 参数名 | 描述 | 示例 |
|---|---|---|
| pageNo | 分页序号 | 1 |
| pageSize | 分页行数 | 10 |
| taskName | 任务名称 | 测试 |
| state | 任务状态 | 0:未完成/1:成功/2:失败 |
| startDate | 开始日期 | 2022-07-01 |
| endDate | 结束日期 | 2022-08-11 |
GET /data-quality/check-result
{
"totalList": [
{
"id": 17,
"processDefinitionId": 36,
"processDefinitionName": "数据质量检查_DWD层",
"processDefinitionCode": 6078547997984,
"processInstanceId": 329,
"processInstanceName": "329-数据质量检查_DWD层-7-20220808021717151",
"projectCode": 6071287478432,
"taskInstanceId": 469,
"taskName": "ONE_ID-空值检查",
"ruleType": 0,
"ruleId": 1,
"checkRuleId": 1,
"ruleName": "(null_check)",
"statisticsValue": 0,
"comparisonValue": 61,
"comparisonType": 0,
"comparisonTypeName": "实际值/期望值 x 100%",
"checkType": 2,
"threshold": 0,
"operator": 3,
"failureStrategy": 0,
"userId": 1,
"userName": "admin",
"state": 1,
"errorOutputPath": "hdfs://bigdata01:9000/user/root/data_quality_error_data/0_329_ONE_ID-空值检查",
"createTime": "2022-08-08 15:17:17",
"updateTime": "2022-08-08 15:17:17"
}
],
"total": 1,
"totalPage": 1,
"pageSize": 10,
"currentPage": 1,
"start": 0
}
查询数据质量检查报告
参数说明
| 参数名 | 描述 | 示例 |
|---|---|---|
| dsName | 检查数据库名称 | hive_ods |
| startDate | 起始日期 | 2021-08-11 14:18:58 |
| endDate | 结束日期 | 2022-08-11 14:18:58 |
| current | 分页序号 | 1 |
| size | 分页行数 | 10 |
- 质量分 趋势
GET /data-quality/check-report/date-score?dsName&startDate&endDate
[
{
"date": "2022-08-08",
"failTaskCount": 0,
"normalResultCount": 0,
"exceptResultCount": 0,
"score": 0
}
]
- 规则数和表覆盖数 趋势
GET /data-quality/check-report/check-rule-date-stat?dsName&startDate&endDate
[
{
"date": "2022-08-08",
"ruleCount": 0,
"relatedTableCount": 0
}
]
- 主题域 质量分
GET /data-quality/check-report/domain-score?startDate&endDate¤t&size
{
"records": [
{
"name": "客户域",
"score": 1,
"ruleCount": 1,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 数据分层 质量分
GET /data-quality/check-report/data-layer-score?startDate&endDate¤t&size
{
"records": [
{
"name": "ODS",
"score": 1,
"ruleCount": 1,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 质量维度 质量分
GET /data-quality/check-report/rule-dim-score?startDate&endDate¤t&size
{
"records": [
{
"ruleId": "1",
"ruleName": "空值校验",
"score": 1,
"ruleCount": 5,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 规则模型 质量分
GET /data-quality/check-report/rule-model-score?startDate&endDate¤t&size
{
"records": [
{
"ruleId": "1",
"ruleName": "空值校验",
"score": 1,
"ruleCount": 5,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 数据库 质量分
GET /data-quality/check-report/database-score?startDate&endDate¤t&size
{
"records": [
{
"name": "hive_ods",
"score": 1,
"ruleCount": 5,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 数据表 质量分
GET /data-quality/check-report/table-score?startDate&endDate¤t&size
{
"records": [
{
"principalId": "datasteward",
"principalName": "数据管家",
"dbName": null,
"tableName": "breweries",
"score": 1,
"ruleCount": 1
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 部门 质量分
GET /data-quality/check-report/dept-score?startDate&endDate¤t&size
{
"records": [
{
"deptId": "-",
"deptName": "-",
"score": 1,
"ruleCount": 1,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}
- 数据责任人 质量分
GET /data-quality/check-report/principal-score?startDate&endDate¤t&size
{
"records": [
{
"principalId": "datasteward",
"principalName": "数据管家",
"score": 1,
"ruleCount": 1,
"tableCount": 0
}
],
"total": 1,
"size": 10,
"current": 1,
"pages": 1
}