Skip to main content

Documentation index: llms.txt. This page is also available as markdown: append .md to this URL or send Accept: text/markdown.

Looker

Extract Looker metadata (users, folders, dashboards, and more) into Catalog using the castor-extractor package.

Prerequisites​

Installation Required

Follow the castor-extractor installation instructions before running the extraction.

  • Get credentials with permissions to call the Looker API

Run Extraction Script​

Once the package has been installed, you should be able to run the following command in your terminal:

castor-extract-looker [arguments]

The script will run and display logs as following:

INFO - Extracting users from Looker API
INFO - Fetched page 1 / 500 results
INFO - Wrote output file: /tmp/catalog/1649079699-users.json
INFO - Extracting folders from Looker API
INFO - Wrote output file: /tmp/catalog/1649079699-folders.json
INFO - Extracting looks from Looker API
INFO - Wrote output file: /tmp/catalog/1649079699-looks.json

...

INFO - Wrote output file: /tmp/catalog/1649079699-projects.json
INFO - Wrote output file: /tmp/catalog/1649079699-summary.json

Credentials​

  • -c, --client-id: Looker API Client ID
  • -s, --client-secret: Looker API Client Secret

Other Arguments​

  • -b, --base-url: Looker base URL
  • -o, --output: Target folder to store the extracted files
  • -t, --timeout: Timeout (in seconds) parameter for Looker API
  • --log-to-stdout: Will write all log outputs to stdout instead of stderr

Specific Export Methods for Looks and Dashboards​

  • --search-per-folder: fetches looks and dashboards folder by folder, one call per folder run in parallel, instead of one paginated search over the whole instance
  • --thread-pool-size: number of parallel API calls, between 1 and 200, defaults to 20

The thread pool drives the per-folder search above, the call that resolves each look to its explore, and the explore fetches. Lower it when extraction puts your Looker instance under more load than you want: fetching an explore compiles its LookML model server-side.

Help

You can also get help with the --help argument.

Use ENV Variables​

If you don't want to specify arguments every time, you can set the following ENV in your .bashrc:

CASTOR_LOOKER_BASE_URL
CASTOR_LOOKER_CLIENT_ID
CASTOR_LOOKER_CLIENT_SECRET

CASTOR_OUTPUT_DIRECTORY
CASTOR_LOOKER_TIMEOUT_SECOND

# To log outputs to `stdout` instead of `stderr`
CASTOR_LOOKER_LOG_TO_STDOUT=TRUE

# To use search_per_folder multithreading
CASTOR_LOOKER_SEARCH_PER_FOLDER=TRUE
CASTOR_LOOKER_THREAD_POOL_SIZE=20

Then the script can be executed without any arguments:

castor-extract-looker

It can also be executed with partial arguments (the script looks in your ENV as a fallback):

castor-extract-looker --output /tmp/catalog