Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

Documentation index: llms.txt. This page is also available as markdown: append .md to this URL or send Accept: text/markdown.

MySQL

Connect Catalog to MySQL to sync metadata from your databases. You can use Catalog-managed credentials or extract metadata for client-managed upload.

Catalog Managed​

Use Catalog-managed connectivity when Catalog holds the credentials and connects to MySQL directly.

Requirements​

You should have:

  • Super-user privileges so you can run administrative commands on MySQL.
  • MySQL version 5.7 or higher.

1. Whitelist Catalog's IP on MySQL​

To allow Catalog to connect to MySQL, you will first need to whitelist Catalog's IP address.

Here are our fixed IPs:

Catalog connects directly to your MySQL databases. As such, it needs to be public. Often, clients give Catalog access to an exact replica of their database.

2. Create a Catalog User on MySQL​

Create the Catalog User​

Use a safe password from a password manager.

CREATE USER 'catalog'@'%' IDENTIFIED BY '<password>';

Grant Rights to the Catalog User​

GRANT SELECT,SHOW VIEW,EXECUTE ON *.* TO 'catalog'@'%';

On MySQL 5.7 you need EXECUTE for certain metadata paths. Newer releases may not require it.

3. Add New Credentials in Catalog App​

Catalog admin required

You must be a Catalog admin to add credentials.

You can now enter the newly created credentials in the Catalog App.

  • Go to Settings > Integrations.
  • Click MySQL Add.
  • Add the credentials using the following format:
{
"host": "host",
"port": "port",
"user": "user",
"password": "password"
}
  • host: MySQL Host
  • port: MySQL Port
  • user: MySQL User
  • password: MySQL Password

Client Managed​

Use client-managed extraction when you generate CSV extracts and upload them yourself.

One-Time Extraction​

For your trial, you can give us a one-shot view of your MySQL metadata. Follow the Warehouse importer specification for the expected format and upload steps.

Use the following queries to extract the necessary information for each file:

  • database.csv
SELECT
CATALOG_NAME AS `id`,
CATALOG_NAME AS `database_name`
FROM INFORMATION_SCHEMA.SCHEMATA
GROUP BY 1, 2
  • schema.csv
SELECT
CONCAT(CATALOG_NAME, ".", SCHEMA_NAME) AS `id`,
CATALOG_NAME AS `database_id`,
SCHEMA_NAME AS `schema_name`,
"" AS `description`,
"" AS `tags`
FROM INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME NOT IN ("INFORMATION_SCHEMA")
  • table.csv
SELECT
CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA, ".", TABLE_NAME) AS `id`,
CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA) AS `schema_id`,
TABLE_NAME AS `table_name`,
TABLE_COMMENT AS `description`,
"" AS `tags`,
CASE
WHEN TABLE_TYPE = "BASE TABLE" THEN "TABLE"
WHEN TABLE_TYPE = "SYSTEM VIEW" THEN "VIEW"
ELSE TABLE_TYPE
END AS `type`
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA NOT IN ("INFORMATION_SCHEMA")
  • column.csv
SELECT
CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA, ".", TABLE_NAME, ".", COLUMN_NAME) AS `id`,
CONCAT(TABLE_CATALOG, ".", TABLE_SCHEMA, ".", TABLE_NAME) AS `table_id`,
COLUMN_NAME AS `column_name`,
COLUMN_COMMENT AS `description`,
CASE
WHEN DATA_TYPE IN ("char", "varchar") THEN "STRING"
WHEN DATA_TYPE IN ("binary", "varbinary", "blob", "tinyblob", "mediumblob", "longblob") THEN "BINARY"
WHEN DATA_TYPE IN ("text", "tinytext", "mediumtext", "longtext") THEN "TEXT"
WHEN DATA_TYPE IN ("enum", "set") THEN "ARRAY"
WHEN DATA_TYPE IN ("bit", "tinyint", "smallint", "mediumint", "int", "bigint", "integer", "long") THEN "INTEGER"
WHEN DATA_TYPE IN ("bool") THEN "BOOLEAN"
WHEN DATA_TYPE IN ("float", "real") THEN "FLOAT"
WHEN DATA_TYPE IN ("double", "double precision", "decimal", "dec", "numeric", "fixed") THEN "DECIMAL"
WHEN DATA_TYPE IN ("date", "datetime", "year") THEN "DATE"
WHEN DATA_TYPE IN ("timestamp") THEN "TIMESTAMP"
WHEN DATA_TYPE IN ("time") THEN "TIME"
WHEN DATA_TYPE IN ("json") THEN "OBJECT"
ELSE UPPER(DATA_TYPE)
END AS `data_type`,
ORDINAL_POSITION AS `ordinal_position`
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA NOT IN ("INFORMATION_SCHEMA")
  • query.csv

Use the following file. We currently cannot retrieve your queries for this integration:

Query

  • view_ddl.csv
SELECT
TABLE_CATALOG AS `database_name`,
TABLE_SCHEMA AS `schema_name`,
TABLE_NAME AS `view_name`,
VIEW_DEFINITION AS `definition`
FROM INFORMATION_SCHEMA.VIEWS
  • user.csv

For MySQL 8.0 and higher

SELECT
USER AS `id`,
CONCAT(USER, "@", HOST) AS `email`
FROM INFORMATION_SCHEMA.USER_ATTRIBUTES
GROUP BY 1, 2

For other versions of MySQL

SELECT
GRANTEE AS `id`,
GRANTEE AS `email`
FROM INFORMATION_SCHEMA.USER_PRIVILEGES
GROUP BY 1, 2

You can upload the resulting files into the Catalog App directly.

After Your Trial​

Generate the 7 files above and push them to our endpoint using the Catalog Uploader.

We will provide you with an API secret and a source_id.

All Files Required

All 7 files are mandatory and data must be consistent and coherent.

Troubleshooting​

This section covers common issues and how to resolve them.

Client-Managed Upload Fails Validation or Ingestion Stays Stale​

Files produced outside the documented CSV shapes do not match what the warehouse importer expects, which leads to rejected uploads or incomplete refreshes even when extraction finishes.

Resolution:

  1. Compare every generated file to the shapes described in Client Managed on this page and to the Warehouse importer specification for column order and required fields.
  2. Regenerate all seven files in one batch so identifiers stay consistent across table.csv, column.csv, and related extracts.
  3. Re-upload through the Catalog Uploader after fixing formats.

Ingestion Errors Reference Parsing or Schema Drift​

MySQL versions differ on catalog tables available for user extracts.

Resolution:

  1. Run the query for the user CSV file that matches your MySQL major version, as listed earlier on this page.
  2. Confirm the query CSV extract follows the placeholder guidance when query logging is not available from MySQL directly.

Catalog-Managed Connection Fails Immediately​

Host allowlisting, wrong host or port, or insufficient grants block login before ingestion starts.

Resolution:

  1. Verify Whitelist Catalog's IP on MySQL includes both fixed IPs for your Catalog hostname.
  2. Confirm the Catalog user still has SELECT, SHOW VIEW, and EXECUTE when required after privilege reviews.

Trial Upload Worked but Production Schedules Fail​

Scheduler credentials or paths often differ between trial laptops and production runners.

Resolution:

  1. Confirm the production job uses the same castor-upload token, source_id, and file paths documented in After Your Trial.
  2. Ensure the upload directory contains only the seven expected files before each push.

Mixed JSON Credentials or Special Characters in Passwords​

Passwords with quotes or backslashes can break JSON payloads copied into the Catalog App.

Resolution:

  1. Paste credentials through a password manager export or escape characters according to JSON rules.
  2. Rotate to a temporary alphanumeric password to isolate quoting issues, then restore your preferred policy.