Log Shipping

To set up the Log Pusher to S3, configure an AWS IAM Role to enable Qwilt to push logs to your Amazon S3 bucket, and then share the Role ARN value with Qwilt.

The following procedure provides step-by-step instructions on how to do this.

1. Log in to your AWS account. Choose All services and search for IAM. Open IAM.

2. In IAM, open the Roles page. Select Create role.

3. Under Trusted entity type, select Custom trust Policy. In the Custom trust policy field, paste the Trusted Policy you received from Qwilt. Below you can find the example to use.

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::052466545929:root"
       },
     "Action": "sts:AssumeRole"
   }
 ]
}

Click Next.

4. On the Add Permissions page, click Next without defining anything. On the Name, review, and create page, enter a Role name and then select Create role.

5. In the IAM Roles page, search for the new role you created and open it.

6. In the Permissions tab, select Add permissions, and then from the drop-down menu, select Create Inline Policy.

7. In the Specify permissions page, click the JSON button. In the Policy editor, enable the following actions:

“putObject”
“putObjectAcl”

We recommend specifying these actions as well, to facilitate troubleshooting if/when needed:

“getObject”
"getObjectAcl"
“listBucket”

Define as Resources the Arn of the S3 bucket you want to share with Qwilt (e.g., "arn:aws:s3:::qwilt_logs"), and specify all objects (e.g., "arn:aws:s3:::qwiltlogs/").

Example:

{
	"Version": "2012-10-17",
	"Statement": [
		{			"Effect":
"Allow",
			"Action": [
				"s3:PutObject",
				"s3:GetObjectAcl",
				"s3:GetObject",
				"s3:ListBucket",
				"s3:PutObjectAcl"
			],
			"Resource": [
				"arn:aws:s3:::qwilt_logs",
"arn:aws:s3:::qwilt_logs/*"
			]
		}
	]
}

8. Click Next.
9. Review the policy. Add a name to the policy that makes sense, and then select Create Policy.
10. In the Role page, under Summary, the Role ARN is displayed.

This ARN value needs to be shared with Qwilt in order to complete the setup.

Copy the value and share it with Qwilt, via a secure method of your choice.

To set up the Log Pusher to GCS, you will need to create a service account and assign it to your GCS bucket with the correct role, and then obtain and share the access object with Qwilt.

The following procedure provides step-by-step instructions on how to do this.

1. Create the Service Account. At this point you will not have to grant it with any permissions. Name the account and click Done.

2. Search for the desired bucket. In this example, the search was filtered for “mdl”.

3. Click the desired bucket, and then click the Permissions tab.

4. Assign the bucket with the relevant service account and then grant it with the Storage Object User role.

5. Your credentials will need to be shared with Qwilt in order to complete the setup. Secrets should be transferred over an encrypted channel (and received secrets will be kept encrypted). Your Qwilt representative will suggest the correct method for sharing your credentials.

Log shipping to Hydrolix is based on a pull API by Hydrolix that pulls the needed transaction logs from a dedicated bucket from within Qwilt Cloud services.

Please contact us at support@qwilt.com to enable this.

Setup the workflow by defining and linking the following in Hydrolix:

• User credentials for accessing Qwilt resources.
• A table to ingest the log data.
• The log fields schema, to specify which of the log fields are parsed and exposed in the table.

STEP 1: Prerequisites

• An active Hydrolix account.
• Qwilt Cloud (QC) Services user credentials. (Provided by Qwilt.)
• JSON file with the needed log fields schema. (Provided by Qwilt.)

STEP 2: Set the QC Services User Credentials in Hydrolix

1. From the Hydrolix navigation bar, select Add New and then choose Credential.

2. In the New credential dialog, define the following:

   • Name - Assign the Credential any name.

   • Description - Any description.

   • Cloud Provider Type - Choose AWS Access Keys.

   • Access Key Id - Enter the Access Key ID provided by Qwilt.

   • Secret Access Key - Enter the Secret Access Key provided by Qwilt.

     

3. Select Create credential.

STEP 3: Create a Table in Hydrolix to Ingest the Log Data

1. From the Hydrolix navigation bar, select Add new and then choose Table.

2. In the New table dialog, select a project and enter a Table name and Description.
   

3. Select Create table.

STEP 4: Define the Log Fields Schema in Hydrolix

1. From the Hydrolix navigation bar, select Add new and then choose Table Transform.

2. In the New ingest transform dialog, define the following:

   • Select table - Select the table you created in the previous step.

   • Transform name - Assign the Table Transform any name.

     

3. Select the Upload method.

4. Browse to and select the JSON file provided by Qwilt.

5. Select Upload Transform.

STEP 5: Create the Ingest Source

1. From the Hydrolix navigation bar, select Add new and then choose Table Source.

2. In the Select Table field, choose the table you created previously.

3. Under Source type, choose Auto Ingest.

   

4. Define the following:

   • Name - Assign any name.
   • Queue Name - The SQS queue name provided by Qwilt.
   • Regex filter - The regex provided by Qwilt.
   • Select transform - Choose Default.
   • Source Credential - Choose the credential you created to allow access to Qwilt resources.
   • Bucket Credential - Choose the credential you created to allow access to Qwilt resources.

5. Select Add source.

Your Qwilt-Hydrolix log shipping is now set up, and you can start reading new transaction logs as they are ingested into the table you created. Data should start arriving within approximately 5 minutes. If you do not see data within this timeframe, please contact support@qwilt.com.

This table presents the full set of MDL fields in the order in which they normally appear in the transaction log. However, your log files may differ.

If your site configuration uses custom URL tokenization, additional MDL fields will be present.

To receive the precise set of metadata fields and their order for your specific log shipping pipeline, please contact us at support@qwilt.com.