Skip to main content

Using dbt-fal with fal cloud

fal cloud is our serverless compute solution that allows you to run Python models on a reliable and scalable infrastructure. Setting up dbt-fal with fal cloud is quick and straightforward.

0. Install fal​

Skip this step, if you already have the latest version of fal installed,

pip install --upgrade fal[cloud]

1. Authenticate to fal cloud​

fal cloud uses GitHub for authentication. Run the following command in your shell:

fal cloud login

Follow the link that's generated and login using GitHub. Come back to the shell, when ready.

2. Generate keys​

Next, generate keys that will allow dbt to connect to fal cloud:

fal cloud generate-keys

This will print a message containing values for KEY_ID and KEY_SECRET. We will need these for setting up the dbt profile.

3. Update your dbt profiles.yml​

In order to run your Python models in fal cloud, you should update the profiles.yml to include the newly generated credentials. Here's an example of how it should look like:

fal_profile:
  target: fal_cloud
  outputs:
    fal_cloud:
      type: fal
      db_profile: db
      host: cloud
      key_secret: MY_KEY_SECRET_VALUE
      key_id: MY_KEY_ID_VALUE
    db:
      type: redshift
      host: MY_REDSHIFT_HOST
      port: MY_REDSHIFT_PORT
      ...

That's it. Doing a dbt run against this profile will execute your Python models in fal cloud.

4. (Optional) Define separate output for fal cloud​

You can have fal outputs, e.g.:

fal_profile:
  target: staging
  outputs:
    staging:
      type: fal
      db_profile: db
    prod:
      type: fal
      db_profile: db
      host: cloud
      key_secret: MY_KEY_SECRET_VALUE
      key_id: MY_KEY_ID_VALUE
    db:
      type: redshift
      host: MY_REDSHIFT_HOST
      port: MY_REDSHIFT_PORT
      ...

In the example above, we have two fal outputs: staging and prod. Output staging will execute your Python models only locally, whereas prod will run them on fal cloud. So now you can control where your models are ran with a -t flag.

This will run Python models locally:

dbt run

And this will run Python models on fal cloud:

dbt run -t prod