counterweight/data-dwh-dbt-project
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merged PR 1330: Stripe UK Source Tests

Browse source 
- [X] The project runs properly with production data.
- [X] The models I have created/edited are fully documented.
- [X] The models I have created/edited contain tests.
- [X] I have checked for DRY opportunities with other models and docs.

This PR adds tests to all the Stripe UK Sources.
This commit is contained in:
Pablo Martín 2024-02-23 11:41:07 +00:00
commit 8d49e9b07b
4 changed files with 267 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

15
dbt_project.yml
View file

@ -1,13 +1,12 @@
# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'dwh_dbt'
version: '1.0.0'
name: "dwh_dbt"
version: "1.0.0"
config-version: 2
# This setting configures which "profile" dbt uses for this project.
profile: 'dwh_dbt'
profile: "dwh_dbt"
# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
@ -19,11 +18,10 @@ seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
clean-targets: # directories to be removed by `dbt clean`
clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"
# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models
@ -40,4 +38,7 @@ models:
+schema: intermediate
reporting:
+materialized: table
+schema: reporting
+schema: reporting
vars:
"dbt_date:time_zone": "Europe/London"

253
models/staging/stripe/_stripe_sources.yml
View file

@ -15,33 +15,104 @@ sources:
- name: id
data_type: character varying
description: "{{ doc('generic_id_desc') }}"
tests:
- unique
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^txn_.{24}$"
- name: fee
data_type: bigint
description: "Fees (in cents) paid for this transaction. Represented as a positive integer when assessed."
tests:
- not_null
- name: net
data_type: bigint
description: "Net impact to a Stripe balance (in cents). A positive value represents incrementing a Stripe balance, and a negative value decrementing a Stripe balance. You can calculate the net impact of a transaction on a balance by amount - fee"
tests:
- not_null
- name: type
data_type: character varying
description: "Transaction type: adjustment, advance, advance_funding, anticipation_repayment, application_fee, application_fee_refund, charge, climate_order_purchase, climate_order_refund, connect_collection_transfer, contribution, issuing_authorization_hold, issuing_authorization_release, issuing_dispute, issuing_transaction, obligation_outbound, obligation_reversal_inbound, payment, payment_failure_refund, payment_network_reserve_hold, payment_network_reserve_release, payment_refund, payment_reversal, payment_unreconciled, payout, payout_cancel, payout_failure, refund, refund_failure, reserve_transaction, reserved_funds, stripe_fee, stripe_fx_fee, tax_fee, topup, topup_reversal, transfer, transfer_cancel, transfer_failure, or transfer_refund. Learn more about balance transaction types and what they represent. To classify transactions for accounting purposes, consider reporting_category instead."
description: "Transaction type: adjustment, advance, advance_funding, anticipation_repayment, application_fee, application_fee_refund, charge, climate_order_purchase, climate_order_refund, connect_collection_transfer, contribution, issuing_authorization_hold, issuing_authorization_release, issuing_dispute, issuing_transaction, obligation_outbound, obligation_reversal_inbound, payment, payment_failure_refund, payment_network_reserve_hold, payment_network_reserve_release, payment_refund, payment_reversal, payment_unreconciled, payout, payout_cancel, payout_failure, network_cost, refund, refund_failure, reserve_transaction, reserved_funds, stripe_fee, stripe_fx_fee, tax_fee, topup, topup_reversal, transfer, transfer_cancel, transfer_failure, or transfer_refund. Learn more about balance transaction types and what they represent. To classify transactions for accounting purposes, consider reporting_category instead."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_in_set:
value_set:
[
"adjustment",
"advance",
"advance_funding",
"anticipation_repayment",
"application_fee",
"application_fee_refund",
"charge",
"climate_order_purchase",
"climate_order_refund",
"connect_collection_transfer",
"contribution",
"issuing_authorization_hold",
"issuing_authorization_release",
"issuing_dispute",
"issuing_transaction",
"obligation_outbound",
"obligation_reversal_inbound",
"payment",
"payment_failure_refund",
"payment_network_reserve_hold",
"payment_network_reserve_release",
"payment_refund",
"payment_reversal",
"payment_unreconciled",
"payout",
"payout_cancel",
"payout_failure",
"network_cost",
"refund",
"refund_failure",
"reserve_transaction",
"reserved_funds",
"stripe_fee",
"stripe_fx_fee",
"tax_fee",
"topup",
"topup_reversal",
"transfer",
"transfer_cancel",
"transfer_failure",
"transfer_refund",
]
- name: amount
data_type: bigint
description: "Gross amount of this transaction (in cents). A positive value represents funds charged to another party, and a negative value represents funds sent to another party."
tests:
- not_null
- name: object
data_type: character varying
description: "Silly column. The value is always `balance_transaction`."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^balance_transaction$"
- name: source
data_type: character varying
description: "This transaction relates to the Stripe object indicated here. Can be different object types (charge, refund, payment, etc)."
- name: status
data_type: character varying
description: "The transaction’s net funds status in the Stripe balance, which are either available or pending."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_in_set:
value_set: ["available", "pending"]
- name: created
data_type: bigint
description: "{{ doc('stripe_created_desc') }}"
tests:
- not_null
- name: currency
data_type: character varying
description: "{{ doc('stripe_currency_desc') }}"
tests:
- not_null
- dbt_expectations.expect_column_value_lengths_to_equal:
value: 3
- name: description
data_type: character varying
description: "An arbitrary string attached to the object. Often useful for displaying to users."
@ -51,6 +122,8 @@ sources:
- name: available_on
data_type: bigint
description: "The date that the transaction’s net funds become available in the Stripe balance."
tests:
- not_null
- name: exchange_rate
data_type: numeric
description: "If applicable, this transaction uses an exchange rate. If money converts from currency A to currency B, then the amount in currency A, multipled by the exchange_rate, equals the amount in currency B. For example, if you charge a customer 10.00 EUR, the PaymentIntent’s amount is 1000 and currency is eur. If this converts to 12.34 USD in your Stripe account, the BalanceTransaction’s amount is 1234, its currency is usd, and the exchange_rate is 1.234."
@ -60,12 +133,30 @@ sources:
- name: reporting_category
data_type: character varying
description: "Read more at https://stripe.com/docs/reports/reporting-categories."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_in_set:
value_set:
[
"fee",
"network_cost",
"charge",
"payout",
"refund",
"topup",
"dispute",
]
- name: _airbyte_raw_id
data_type: character varying
description: "{{ doc('_airbyte_raw_id_desc') }}"
- name: _airbyte_extracted_at
data_type: timestamp with time zone
description: "{{ doc('_airbyte_extracted_at_desc') }}"
tests:
- not_null
- dbt_expectations.expect_row_values_to_have_recent_data:
datepart: day
interval: 2
- name: _airbyte_meta
data_type: jsonb
description: "{{ doc('_airbyte_meta_desc') }}"
@ -79,21 +170,35 @@ sources:
- name: id
data_type: character varying
description: "{{ doc('generic_id_desc') }}"
tests:
- unique
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^(ch|py)_.{24}$"
- name: card
data_type: jsonb
description: "{{ doc('field_docs_na_desc') }}"
- name: paid
data_type: boolean
description: "true if the charge succeeded, or was successfully authorized for later capture."
tests:
- not_null
- name: order
data_type: character varying
description: "{{ doc('field_docs_na_desc') }}"
- name: amount
data_type: bigint
description: "Amount intended to be collected by this payment. A positive integer representing how much to charge in the smallest currency unit (e.g., 100 cents to charge $1.00 or 100 to charge ¥100, a zero-decimal currency). The minimum amount is $0.50 US or equivalent in charge currency. The amount value supports up to eight digits (e.g., a value of 99999999 for a USD charge of $999,999.99)."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
- name: object
data_type: character varying
description: "Silly column. The value is always `charge`."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^charge$"
- name: review
data_type: character varying
description: "ID of the review associated with this charge if one exists."
@ -103,9 +208,15 @@ sources:
- name: status
data_type: character varying
description: "The status of the payment is either succeeded, pending, or failed."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_in_set:
value_set: ["succeeded", "pending", "failed"]
- name: created
data_type: bigint
description: "{{ doc('stripe_created_desc') }}"
tests:
- not_null
- name: dispute
data_type: character varying
description: "{{ doc('field_docs_na_desc')}}"
@ -121,12 +232,20 @@ sources:
- name: updated
data_type: bigint
description: "{{ doc('stripe_created_desc') }}"
tests:
- not_null
- name: captured
data_type: boolean
description: "If the charge was created without capturing, this Boolean represents whether it is still uncaptured or has since been captured."
tests:
- not_null
- name: currency
data_type: character varying
description: "{{ doc('stripe_currency_desc') }}"
tests:
- not_null
- dbt_expectations.expect_column_value_lengths_to_equal:
value: 3
- name: customer
data_type: character varying
description: "ID of the customer this charge is for if one exists."
@ -142,6 +261,8 @@ sources:
- name: refunded
data_type: boolean
description: "Whether the charge has been fully refunded. If the charge is only partially refunded, this attribute will still be false."
tests:
- not_null
- name: shipping
data_type: jsonb
description: "Shipping information for the charge."
@ -178,9 +299,17 @@ sources:
- name: payment_intent
data_type: character varying
description: "ID of the PaymentIntent associated with this charge, if one exists."
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^pi_.{24}$"
- name: payment_method
data_type: character varying
description: "ID of the payment method used in this charge."
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^pm_.{24}$"
- name: receipt_number
data_type: character varying
description: "This is the transaction number that appears on email receipts sent for this charge. This attribute will be null until a receipt has been sent."
@ -190,9 +319,19 @@ sources:
- name: amount_captured
data_type: bigint
description: "Amount in cents captured (can be less than the amount attribute on the charge if a partial capture was made)."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: False
- name: amount_refunded
data_type: bigint
description: "Amount in cents refunded (can be less than the amount attribute on the charge if a partial refund was issued)."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: False
- name: application_fee
data_type: character varying
description: "The application fee (if any) for the charge. See the Connect documentation for details."
@ -208,6 +347,10 @@ sources:
- name: balance_transaction
data_type: character varying
description: "ID of the balance transaction that describes the impact of this charge on your account balance (not including refunds or disputes)."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^txn_.{24}$"
row_condition: "balance_transaction is not null"
- name: statement_descriptor
data_type: character varying
description: "For card charges, use statement_descriptor_suffix instead. Otherwise, you can use this value as the complete description of a charge on your customers’ statements. Must contain at least one letter, maximum 22 characters."
@ -235,6 +378,11 @@ sources:
- name: _airbyte_extracted_at
data_type: timestamp with time zone
description: "{{ doc('_airbyte_extracted_at_desc') }}"
tests:
- not_null
- dbt_expectations.expect_row_values_to_have_recent_data:
datepart: day
interval: 2
- name: _airbyte_meta
data_type: jsonb
description: "{{ doc('_airbyte_meta_desc') }}"
@ -250,12 +398,24 @@ sources:
- name: id
data_type: character varying
description: "{{ doc('generic_id_desc') }}"
tests:
- unique
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^pi_.{24}$"
- name: amount
data_type: bigint
description: "Amount intended to be collected by this PaymentIntent. A positive integer representing how much to charge in the smallest currency unit (e.g., 100 cents to charge $1.00 or 100 to charge ¥100, a zero-decimal currency). The minimum amount is $0.50 US or equivalent in charge currency. The amount value supports up to eight digits (e.g., a value of 99999999 for a USD charge of $999,999.99)."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
- name: object
data_type: character varying
description: "Silly column. The value is always `payment_intent`."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^payment_intent$"
- name: review
data_type: character varying
description: "ID of the review associated with this PaymentIntent, if any."
@ -265,21 +425,43 @@ sources:
- name: status
data_type: character varying
description: "Status of this PaymentIntent, one of requires_payment_method, requires_confirmation, requires_action, processing, requires_capture, canceled, or succeeded. Read more about each PaymentIntent status."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_in_set:
value_set:
[
"requires_payment_method",
"requires_confirmation",
"requires_action",
"processing",
"requires_capture",
"canceled",
"succeeded",
"requires_action",
]
- name: charges
data_type: jsonb
description: "{{ doc('field_docs_na_desc') }}"
- name: created
data_type: bigint
description: "{{ doc('stripe_created_desc') }}"
tests:
- not_null
- name: invoice
data_type: character varying
description: "ID of the invoice that created this PaymentIntent, if it exists."
- name: updated
data_type: bigint
description: "{{ doc('stripe_created_desc')}}"
tests:
- not_null
- name: currency
data_type: character varying
description: "{{ doc('stripe_currency_desc') }}"
tests:
- not_null
- dbt_expectations.expect_column_value_lengths_to_equal:
value: 3
- name: customer
data_type: character varying
description: |
@ -294,6 +476,8 @@ sources:
- name: metadata
data_type: jsonb
description: "{{ doc('stripe_metadata_desc') }}"
tests:
- not_null
- name: shipping
data_type: jsonb
description: "Shipping information for the charge."
@ -326,6 +510,10 @@ sources:
- name: latest_charge
data_type: character varying
description: "The latest charge created by this PaymentIntent."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^(ch|py)_.{24}$"
row_condition: "latest_charge is not null"
- name: receipt_email
data_type: character varying
description: "Email address that the receipt for the resulting payment will be sent to. If receipt_email is specified for a payment in live mode, a receipt will be sent regardless of your email settings."
@ -347,9 +535,19 @@ sources:
- name: amount_received
data_type: bigint
description: "Amount that this PaymentIntent collects."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: False
- name: amount_capturable
data_type: bigint
description: "Amount that can be captured from this PaymentIntent."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
strictly: False
- name: last_payment_error
data_type: jsonb
description: "The payment error encountered in the previous PaymentIntent confirmation. It will be cleared if the PaymentIntent is later updated for any reason."
@ -397,6 +595,11 @@ sources:
- name: _airbyte_extracted_at
data_type: timestamp with time zone
description: "{{ doc('_airbyte_extracted_at_desc') }}"
tests:
- not_null
- dbt_expectations.expect_row_values_to_have_recent_data:
datepart: day
interval: 2
- name: _airbyte_meta
data_type: jsonb
description: "{{ doc('_airbyte_meta_desc') }}"
@ -409,36 +612,73 @@ sources:
- name: id
data_type: character varying
description: "{{ doc('generic_id_desc') }}"
tests:
- unique
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^(pyr_.{24}|re_.{24})$"
- name: amount
data_type: bigint
description: "Amount, in cents."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_between:
min_value: 0
- name: charge
data_type: character varying
description: "ID of the charge that’s refunded."
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^(ch|py)_.{24}$"
- name: object
data_type: character varying
description: "Silly column. The value is always `refund`."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^refund$"
- name: reason
data_type: character varying
description: "Reason for the refund, which is either user-provided (duplicate, fraudulent, or requested_by_customer) or generated by Stripe internally (expired_uncaptured_charge)."
- name: status
data_type: character varying
description: "Status of the refund. This can be pending, requires_action, succeeded, failed, or canceled. Learn more about failed refunds."
tests:
- not_null
- dbt_expectations.expect_column_values_to_be_in_set:
value_set:
[
"succeeded",
"pending",
"failed",
"requires_action",
"canceled",
]
- name: created
data_type: bigint
description: "{{ doc('stripe_created_desc') }}"
tests:
- not_null
- name: updated
data_type: bigint
description: "{{ doc('stripe_created_desc') }}"
- name: currency
data_type: character varying
description: "{{ doc('stripe_currency_desc') }}"
tests:
- not_null
- dbt_expectations.expect_column_value_lengths_to_equal:
value: 3
- name: metadata
data_type: jsonb
description: "{{ doc('stripe_metadata_desc') }}"
- name: payment_intent
data_type: character varying
description: "ID of the PaymentIntent that’s refunded."
tests:
- not_null
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^pi_.{24}$"
- name: receipt_number
data_type: character varying
description: "This is the transaction number that appears on email receipts sent for this refund."
@ -447,7 +687,11 @@ sources:
description: "This refers to the transfer reversal object if the accompanying transfer reverses. This is only applicable if the charge was created using the destination parameter."
- name: balance_transaction
data_type: character varying
description: "Balance transaction that describes the impact on your account balance."
description: "Balance transacion that describes the impact on your account balance."
tests:
- dbt_expectations.expect_column_values_to_match_regex:
regex: "^txn_.{24}$"
row_condition: "balance_transaction is not null"
- name: destination_details
data_type: jsonb
description: "Transaction-specific details for the refund."
@ -460,6 +704,11 @@ sources:
- name: _airbyte_extracted_at
data_type: timestamp with time zone
description: "{{ doc('_airbyte_extracted_at_desc') }}"
tests:
- not_null
- dbt_expectations.expect_row_values_to_have_recent_data:
datepart: day
interval: 2
- name: _airbyte_meta
data_type: jsonb
description: "{{ doc('_airbyte_meta_desc') }}"

6
package-lock.yml
View file

@ -1,6 +1,10 @@
packages:
- package: dbt-labs/codegen
version: 0.12.1
- package: calogica/dbt_expectations
version: 0.9.0
- package: dbt-labs/dbt_utils
version: 1.1.1
sha1_hash: acdde602cd2e228b1dac1cf1a7e5f98f12c16b3b
- package: calogica/dbt_date
version: 0.8.1
sha1_hash: 0d8e1bf7188f14d46813097c65818e3f96237266

4
packages.yml
View file

@ -1,3 +1,5 @@
packages:
- package: dbt-labs/codegen
version: 0.12.1
version: 0.12.1
- package: calogica/dbt_expectations
version: [">=0.9.0", "<0.10.0"]