Log Shipping
  • 11 Dec 2024
  • 10 Minutes to read
  • PDF

Log Shipping

  • PDF

Article summary

Introduction

Qwilt keeps track of its HTTP transactions via transaction logs. These logs can be used for auditing, performance analysis, or debugging purposes - and are collected across the entire Qwilt deployment. You can adjust the transaction logs’ output to suit your needs.

Qwilt can send these logs in one or more of the following ways:

In addition to the endpoint, you can:

  • Choose the requested file format of the logs - TSV/JSON.
  • Choose the fields in the files (see the lists of possible fields in the tables below).
  • Filter by Delivery Service (for Content Providers), or by response code class (2xx/3xx/4xx/5xx).

Setting the Endpoint

Click the links below to expand and learn more about each option.

AWS/S3The Log Pusher to S3 automatically sends the logs to service providers and content providers who are enrolled in a log delivery service for their service or deployment.

To set up the Log Pusher to S3, configure an AWS IAM Role to enable Qwilt to push logs to your Amazon S3 bucket, and then share the Role ARN value with Qwilt.

The following procedure provides step-by-step instructions on how to do this.

  1. Log in to your AWS account. Choose All services and search for IAM. Open IAM.

DLAWSIAM.png

  1. In IAM, open the Roles page. Select Create role.

IAM Roles.png

  1. Under Trusted entity type, select Custom trust Policy. In the Custom trust policy field, paste the Trusted Policy you received from Qwilt. Below you can find the example to use.

DLCustomTrustPolicy.png

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::052466545929:root"
       },
     "Action": "sts:AssumeRole"
   }
 ]
}

Click Next.

  1. On the Add Permissions page, click Next without defining anything. On the Name, review, and create page, enter a Role name and then select Create role.

DLRoleName.png

  1. In the IAM Roles page, search for the new role you created and open it.

  2. In the Permissions tab, select Add permissions, and then from the drop-down menu, select Create Inline Policy.

DLCreateInlinePolicy.png

  1. In the Specify permissions page, click the JSON button. In the Policy editor, enable the following actions:
“putObject”
“putObjectAcl”

We recommend specifying these actions as well, to facilitate troubleshooting if/when needed:

“getObject”
"getObjectAcl"
“listBucket”

Define as Resources the Arn of the S3 bucket you want to share with Qwilt (e.g., "arn:aws:s3:::qwilt_logs"), and specify all objects (e.g., "arn:aws:s3:::qwiltlogs/").

Example:

{
	"Version": "2012-10-17",
	"Statement": [
		{			"Effect":
"Allow",
			"Action": [
				"s3:PutObject",
				"s3:GetObjectAcl",
				"s3:GetObject",
				"s3:ListBucket",
				"s3:PutObjectAcl"
			],
			"Resource": [
				"arn:aws:s3:::qwilt_logs",
"arn:aws:s3:::qwilt_logs/*"
			]
		}
	]
}
  1. Click Next.
  2. Review the policy. Add a name to the policy that makes sense, and then select Create Policy.
  3. In the Role page, under Summary, the Role ARN is displayed.

This ARN value needs to be shared with Qwilt in order to complete the setup.

Copy the value and share it with Qwilt, via a secure method of your choice.

PusherFinal.png

GCSThe Log Pusher to GCS bucket automatically sends the logs to service providers and content providers who are enrolled in a log delivery service for their service or deployment. Transaction logs are pushed as soon as they are created; however, in the event of heavy traffic or communication issues, they may arrive up to 15 minutes later.

To set up the Log Pusher to GCS, you will need to create a service account and assign it to your GCS bucket with the correct role, and then obtain and share the access object with Qwilt.

The following procedure provides step-by-step instructions on how to do this.

  1. Create the Service Account. At this point you will not have to grant it with any permissions. Name the account and click Done.

GCS1.png

  1. Search for the desired bucket. In this example, the search was filtered for “mdl”.

GCS2.png

  1. Click the desired bucket, and then click the Permissions tab.

GCS3.png

  1. Assign the bucket with the relevant service account and then grant it with the Storage Object User role.

NewRole.png

GCSStep4Newb.png

  1. Your credentials will need to be shared with Qwilt in order to complete the setup. Secrets should be transferred over an encrypted channel (and received secrets will be kept encrypted). Your Qwilt representative will suggest the correct method for sharing your credentials.

GCSStep5New.png

DatadogTransaction logs can be shipped to Datadog via Datadog API. To enable this, contact us at support@qwilt.com.
HydrolixTransaction logs can be shipped to your Hydrolix account. To enable this, contact us at support@qwilt.com.

Setting the Log Shipping File Format

When shipped into a cloud bucket (either S3 or GCS), the transaction logs can be stored as either a TSV (tab-separated values) or a JSON file, compressed in gzip format.

Note: Logs are created every five minutes, or as soon as the log contains 50MB of data - whichever comes first. Transaction logs are pushed as soon as they are created; however, in the event of heavy traffic or communication issues, they could arrive up to 24 hours later.

The transaction logs’ file naming convention is as follows:

<file start timestamp(YYYYMMDD-HHmmSS)>.<service-name>.<unique Id>.log.gz

For example:

20230801-055130.example-service.03495.log.gz

Log Shipping MDL Field Descriptions

The following table describes each of the parameters, in the order in which they appear in the transaction log. Note that when a parameter value is not available, a hyphen (-) appears instead.

Note

For large transactions (also known as Content Range Requests or segmented transactions), Qwilt's log shipping defaults to transferring only the client transactions, referred to as 'main transactions.' These main transaction records aggregate information from all slices. Each main transaction log details the total volume served (sentBytes), the total volume read from the Origin (fetchedBytes), and the duration of all slices/sub-transactions (durationMilli). The NumSliceSubTrx field indicates the number of slices used, while NumSliceMisses represents the slices that were missed.

Disclaimer

This section describes the full set of MDL fields. However, the fields included in your MDL files may vary based on the version of the Qwilt CDN serving your traffic (CP) or in your network (SP). Additionally, the order of fields in the log data may differ.

To receive the precise set of metadata fields and their order for your specific log shipping pipeline, please contact us at support@qwilt.com.

Global Fields
FieldData TypeExample Value(s)Description
startTimeString13-04-23 18:17:02.883Unixtime of the record creation time. Date and time.
startTimeEpochMilliInteger1681409822883The record creation time. The amount of milliseconds since the epoch.
durationMilliInteger52Transaction duration in milliseconds.
sourceStringdynamic-cdn-extentionIndicates the source of the response.

Valid Values:
dynamic-cdn-extension - The content was provided by an origin server.
cache-delivery - The content was provided by the Qwilt CDN cache.
self-generated - The response was generated by the Qwilt CDN cache. This value only applies to HTTP error responses.
siteNameStringc7081-acme-liveAn internal ID assigned by Qwilt to your site. This is also known as Delivery Service.
clientIpString: IPv4 or IPv6 address$0:1$7PLOwpcQM7p789G4pWAw3+KvaPA= (This is an example of an obfuscated IP.)Client side IP - either as clear text or obfuscated.
clientPortInteger49222The client-side TCP port.
serverIpString: IPv4 or IPv6 address211.94.171.32The IP address of the CDN cache, from which the content was delivered.
serverPortInteger443The server-side TCP port.
sentBytesInteger853222(Represents the MDL field L7Goodput). The number of L7 bytes transferred to the client (including both content and HTTP response headers).
httpResponseCodeInteger200The HTTP response code value.
requestRangeStringbytes=795346685-795548657The HTTP range.
responseRangeStringbytes 795346685-795548657/986715125The HTTP content range.
userAgentStringMozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Mobile/15E148 Safari/604.1The User-Agent, if the user-agent header was included in the request.
altDeviceNameStringN/ARequest the device name header.
origServerIpString100.100.100.100The origin’s server IP.
trxStatusStringacceptedValid Values:
accepted - the transaction was served.
rejected - the transaction was rejected.
trxStatusReasonStringphysical-capacity-exceededProvides more detail about the transaction status.

Valid Values:
‘-’ when the transaction was accepted.
physical-capacity-exceeded - The transaction was rejected because the link capacity was reached.
efficiency-policy - Transaction was rejected because resources are dedicated to prioritized transactions.
set-limit-exceeded - Transaction was rejected because a configured global limit was reached.
refererStringhttps://www.geeksforgeeks.org/The Referer HTTP request header value.
httpMethodStringgetThe HTTP method used in the transaction. Valid Values: GET HEAD OPTIONS
fetchedBytesString853393(Represents the MDL field L7UpstreamGoodput). The size of the HTTP content payload plus response headers for the upstream transaction.
httpVersionString1.1The HTTP version.
triggerStringexternalValid Values:
external - An external client initiated the transaction.
cdn - The CDN triggered the transaction.
originHostNamesStringorigin1.com,origin2.com,origin3.com

In this example, we can see that origin1.com was contacted first, and then origin2.com was contacted, and then origin3.com was contacted.
This field is relevant when the ‘source’ is 'dynamic-cdn-extension.' A comma-separated list of the origin servers contacted by the CDN while handling the transaction.
originHttpResponseCodesString503,-,200 (In this example, taken together with the example values for originHostNames, we can see that origin1.com responded with 503, origin2.com did not respond, and origin3.com responded with 200).This field is relevant when the ‘source’ is 'dynamic-cdn-extension.' A comma-separated list of the response codes received from each of the origins listed in the ‘originHostNames’ field.
CacheStatusStringhitDescribes how the transaction was handled by the cache.

Valid Values:
hit: The content was served from the local cache.
miss: The content was served from the origin server, because it was not available in the cache.
bypass: The content was served from the origin server, even though it was available in the cache, due to configuration or specific request information.
expired: The content was served from the origin server, because the content in the cache was stale.
stale: Stale content was served from the local cache. For example, if the origin server is unavailable and the configuration rules enable serving stale content when the origin server is unavailable.
updating: Stale content was served from the local cache, while a background request was sent to the origin server to update it.
revalidated: Stale content was revalidated against the origin server and served from the local cache, as it was still up-to-date.
deliveryHopCountInteger1Contains the position of this cache in the chain of caches, from the Edge server to origin (including the Edge). The position always starts from 1, where 1 indicates that this cache is the Edge.
deliveryChainLengthInteger2Contains the number of CDN servers in the chain of caches from the Edge server to origin (including the Edge). Chain length is always at least 1.
transactionChainIdString10000000397c92e9-88108992-592For transactions delivered by the OCN, a string that uniquely identifies this transaction chain.
TTFBInteger300Time (in milliseconds), from when the OCN received the request until it sends the first byte of response.
ClientASNInteger23456ASN number of the client IP.
ClientCountryCodeStringUKCountryCode of the client IP.
dataTypeStringContentEither 'content' or 'router-proxy'.
downstreamTypeStringPeerValid Values:
'Peer': north-south.
'child': east-west.
'internal' if sub-transaction.
'-' if the downstream was a client.
TriggerTrxIdString10000004776d9176-3-1Client transaction ID.
subTrxTypeStringSliceThe transaction type.

Valid Values:
Slice: indicating that this is a slice/ sub-transaction.
Other: Non slice.
sliceRangeString1024-1535For a triggered sub-transaction of the type slice, this field holds the range that the sub-transaction is in charge of.
cdnCacheStatusStringMissFor client-facing transactions that were sliced. If this transaction - or any sub-transaction of it - had to fetch content from the origin, then this holds as ‘miss’.
edgeLocationStringUKThe CDN location country code.
x-forward-forString203.0.113, 198.51.100, 192.0.2The original IP address (presented as subnet) of the client connecting through one or more proxies.
originTTFBInteger200Time (in milliseconds) from when the request was sent to the origin until the first byte of response was received from it.
CmcdSidString6e2fb550—c457—11e9—bb97—0800200c9a66Common Media Client Data (CMCD) Session ID.
NumSliceSubTrxNumber20For client-facing large transactions. It holds the amount of slices and sub-transactions that were triggered.
NumSliceMissesNumber1For client-facing large transactions. It holds the amount of slices and sub-transactions that had a type of miss, bypass, or expired. Otherwise, it holds 0.
topTierCacheStatusStringContains a string that describes how the transaction was handled by the upper-most cache in the chain of caches that handled this transaction. Contains '-' if no other QN was involved in the transaction.
subnetString83.219.163.0/24Contains the client IP address, partially masked as a subnet.
responseRttMsecString42Socket-measured RTT (millisecond).
contentTypeStringtext/html;charset=utf—8Content-Type' HTTP header of the response.
SSL-CipherStringTLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256The selected SSL cipher suite.
Fields Relevant Only to Content Providers
FieldData TypeExample Value(s)Description
uriStringhttp://qb1.my—cdn.com/some/pathThe client URL.
Fields Relevant Only to Service Providers
FieldData TypeExample Value(s)Description
downstreamSystemIdStringThe System ID of the downstream system.
upstreamSystemIdStringThe System ID of the upstream system.

Was this article helpful?

What's Next
Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.