Spring boot Integration to AWS Lake

Hi @vani.nithya.indla,

Just to let you know that the updated AWS Lake Formation integration (having version 1.0.3) has been released and is available on Collibra’s Marketplace.

Hi @spring-team.collibra.com

Thank you very much !!!

We will test and get back for any queries.

Hi @spring-team.collibra.com

Thanks for the update

We have few queries on the package shared

1. Two grant workflows are shared
   1. Grant Tag Permissions
   2. Grant Permissions resource
   Will you be sharing revoke workflows separately ? Also there is a Grant/ revoke batch permission workflow as per the flow diagram , will you be developing those workflows as well?
   
   
   

2. In Grant permissions resource workflow shared, we need to call Request Asset Access workflow , but when we call the workflow from data basket, it is directly going to approval without calling ARN attach / Grant AWS workflows and in the logs we could fine the below

We are selecting column ingested from AWS but still getting this error. Can you help us on this?

Hi @spring-team.collibra.com

Thanks for your update , We will test it and get back to you.

We are testing the Grant Tag permissions workflow and found that after selecting ARN, while selecting LF-Tag key we are not getting task instead it is just showing the below screen with done option.

There is no pop up task to select the key or value. Can you please advise us on this?

Hi @vani.nithya.indla,
Thanks again for your feedback.
Can you provide us with a screenshot of the ‘view workflow diagram option’ and the logs, please?
Also, what is your Collibra version and is it a cloud or an on-premise instance?

Thanks

Hi @spring-team.collibra.com
I would like to raise an issue with the bidirectional integration

1. When we ingest data from AWS to Collibra in Read only domains, Database has LF Tag relation , Table hasLF tag relation and Column has LF Tag relation should be populated

But if in AWS Lake formation, if the LF - Tag is inherited from Database to Table / Table to Column, The inherited LF-Tag relation is not ingested

For Example in AWS I have a database X with Table a and Table B, and I have assigned LF-Tag on Database X which inherited to Table a and Table B

When Data is ingested in Collibra Only Database X has LF tag relation is getting ingested but Table a and Table b doesnt have that relation populated.

This is very important to classify the data for our custom needs. Please let us know your thoughts on this.

Hi @spring-team.collibra.com

Suddenly we are getting the below error while syncing the results . Can you please help us to decode this error:

2022-08-08 12:35:41,291 [scheduling-1] ERROR com.collibra.aws.lakeformation.scheduler.ScheduledSyncProcess - Sync Scheduler Error {“message”:“An unexpected error occurred during execution.”,“details”:“Error reading entity from input stream.”,“cause”:“Unrecognized field “result” (class com.collibra.internal.api.model.output.Job), not marked as ignorable (17 known properties: “userId”, “createdOn”, “resourceType”, “endDate”, “name”, “system”, “startDate”, “state”, “visibility”, “cancelable”, “lastModifiedBy”, “type”, “id”, “lastModifiedOn”, “createdBy”, “progressPercentage”, “message”])\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 376] (through reference chain: com.collibra.internal.api.model.output.Job[“result”])”}

Hi @spring-team.collibra.com

Thanks a lot for your support

For sbi-aws-lakeformation-v1.0.2 core as collibra-integration-library -1.1.6 (

com.collibra.internal.api.model.out put doesn’t have attribute result

) but still ingesting and syncing worked without any issue.

sbi-aws-lakeformation-v1.0.3 now added dependency core as collibra-integration-library -1.1.8( com.collibra.internal.api.model.output have attribute result) and working without job exception.

Is there anyway can check in collibra end or is there any update happened on the job level? on JOB API like response format or attribute changes

https://tfnsw-dev.collibra.com/rest/2.0/Jobs

Hi @spring-team.collibra.com

Is it possible to organize a 30 mins session to discuss on Grant LF-Tag workflow
We are receiving below errors as per the workflow inputs

Please let us know on this.

Hi @spring-team.collibra.com

In v 1.0.2 we haven’t received any workflows.

With the latest version 1.0.3 we have received 3 workflows of which we are testing “Grant AWS Permissions (LF-TAG).bar”

When we triggered the workflow it is creating 4 asset types

1. AWS Entry
   Attributes:
   Resource Type : Default value “TAG”
   Grantable-Permissions : Default Value “SELECT”
   Tag-Permissions : Default Value “DESCRIBE”
   ARN
   Catalog ID
   Tag Key
   
   
   
   
   
   
   

2. Aws-Grant-Batch-Permissions-Tag
   Attributes:
   ARN
   Catalog ID
   Tag Key
   Tag Value
   Has aws-entry
   
   
   
   
   
   

3. AWS- LF-Tag Key
   has Tag-value
   

4.AWS-LF-Tag Value

Also in the workflow both select and describe are coming up and it is so confusing to understand the LF Tag grant functionality

Hi @spring-team.collibra.com Can we please schedule a call to identify the issue , We require to check this functionality to collaborate with collibra team to raise our customizations if required.

Hi @vani.nithya.indla,

We will check regarding the 30-minute session and keep you updated. In the meantime, could you please provide us with the Spring Boot logs (via a private message) for each synchronisation? Thanks

Hi @spring-team.collibra.com We are good if session is delayed, Can you please provide us some resolution for the issue raised, I have attached the workflow code as well

Hi @spring-team.collibra.com

We have a query on leaveraging Lineage harvester feature for this integration, Is there a way to load transformations from Glue jobs ? Can you please suggest us on any approach.

Hi @spring-team.collibra.com

We noticed that while ingesting data from AWS lake formation to Collibra, partition columns are skipped and not getting ingested into Collibra.

Can you please let us know on this.