FlowObjects Data Loader for Snowflake - command line parameters

FlowObjects Data Loader

Command Line Parameters

Purpose

The FlowObjects Data Loader is a command line interface (CLI) tool that is designed to efficiently load data to Snowflake. Some of the features include:

  • reduce upload time by leveraging compression
  • break up large files into smaller optimal chunks
  • produce logs to console, file, or a table to audit a data load
  • control auto commit
  • add column(s) and value(s), i.e. meta-data - - inline, without modifying the source
  • easily embed in your existing processes (i.e. automation scripts)

See also: Using CLI switches or environment variables

Data Loader 3.0.0, CLI switches and flags usage

usage: flowobjects_data_loader [-h] [-v] [-s] --file FILE --sf_account
                               SF_ACCOUNT --sf_username SF_USERNAME
                               --sf_password SF_PASSWORD --sf_database
                               SF_DATABASE --sf_warehouse SF_WAREHOUSE
                               --sf_role SF_ROLE --sf_schema SF_SCHEMA
                               --sf_table SF_TABLE --sf_file_format
                               SF_FILE_FORMAT --sf_stage_type SF_STAGE_TYPE
                               [--sf_autocommit SF_AUTOCOMMIT]
                               [--sf_stage_name SF_STAGE_NAME]
                               [--encoding ENCODING] [--delimiter DELIMITER]
                               [--header_rows HEADER_ROWS]
                               [--on_error ON_ERROR]
                               [--transaction_name TRANSACTION_NAME]
                               [--error_table ERROR_TABLE]
                               [--split_file_size SPLIT_FILE_SIZE]
                               [--transformation_file TRANSFORMATION_FILE]
                               [--copy_into COPY_INTO]
                               [--additional_headers [ADDITIONAL_HEADERS [ADDITIONAL_HEADERS ...]]]
                               [--additional_content [ADDITIONAL_CONTENT [ADDITIONAL_CONTENT ...]]]
                               [--no_purge] [--remove_split_dir] [--log]
                               [--log_to_error_table] [--quiet]
                               [--synchronous] [--split_file]
                               [--use_upload_dir] [--no_error_on_empty]
                               [--sf_no_warehouse_resume] [--sf_no_compress]
                               [--show_timing]

The arguments below are divided into switches for which a value must be
provided and flags for which no value should be provided. Most switches can
be used as environment variables instead. Some switches are required while
others are optional. Switches must be lowercase and are shown below with --
in front of them. Environment variable equivalents are shown next to the
switch in all uppercase. For more details, visit:
https://community.flowobjects.com/t/using-cli-switches-or-environment-
variables

arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -s, --sysinfo

required switches (requires value(s) for switch):
  --file FILE           File(s) or Directory/Directories to Upload (i.e.
                        file: mydata.csv || files: *.csv || directory of
                        files: myfilesdir/). If provided file not already
                        compressed, will automatically compress (in GZIP
                        format) before uploading (see --sf_no_compress flag
                        not to automatically compress). Already compressed
                        supported file formats will pass through without
                        being compressed again.
  --sf_account SF_ACCOUNT
                        Snowflake account name.
  --sf_username SF_USERNAME
                        Snowflake user name.
  --sf_password SF_PASSWORD
                        Snowflake password.
  --sf_database SF_DATABASE
                        Snowflake database.
  --sf_warehouse SF_WAREHOUSE
                        Snowflake warehouse.
  --sf_role SF_ROLE     Snowflake role.
  --sf_schema SF_SCHEMA
                        Snowflake schema.
  --sf_table SF_TABLE   Snowflake table to place file data in (i.e.
                        DATABASE.SCHEMA.TABLENAME).
  --sf_file_format SF_FILE_FORMAT
                        File format of files to upload (i.e.
                        EXCEL_DEMO_FORMAT).
  --sf_stage_type SF_STAGE_TYPE
                        Stage type (i.e. TABLE, NAMED, USER).

optional switches (requires value(s) for switch):
  --sf_autocommit SF_AUTOCOMMIT
                        Control whether on connection close all changes are
                        automatically committed [default: True]. Set to True
                        to enable autocommit or False to disable it.
  --sf_stage_name SF_STAGE_NAME
                        Stage name if using a named or table stage (i.e.
                        DATABASE.SCHEMA.STAGENAME) [default: None]
  --encoding ENCODING   Encoding for file to upload [default: None]. If not
                        specified, read from Snowflake file format.
  --delimiter DELIMITER
                        Line delimiter of file to upload [default: None]. If
                        not specified, read from Snowflake file format. Not
                        applicable to JSON file type.
  --header_rows HEADER_ROWS
                        Number of header rows to skip [default: 0]. Not
                        applicable to JSON file type.
  --on_error ON_ERROR   Setting for 'COPY INTO' command if an error is
                        encountered (i.e. CONTINUE, SKIP_FILE,
                        SKIP_FILE_, SKIP_FILE_, ABORT_STATEMENT).
                        [default: CONTINUE].
  --transaction_name TRANSACTION_NAME
                        Transaction name [default: FILE_UPLOAD]
  --error_table ERROR_TABLE
                        Table to write errors to (i.e.
                        DATABASE.SCHEMA.ERRORTABLENAME). If not specified,
                        will default to ERROR_TABLE and must be usedtogether
                        with --log_to_error_table
  --split_file_size SPLIT_FILE_SIZE
                        Size of split record files in bytes. [default:
                        100000000 (100 MB)]. Not applicable to JSON file type
                        or pre-compressed file.
  --transformation_file TRANSFORMATION_FILE
                        JSON file specifying transformations. [default: None]
  --copy_into COPY_INTO
                        JSON file with custom COPY INTO statement to be used
                        instead of auto-generated default. [default: None]
  --additional_headers [ADDITIONAL_HEADERS [ADDITIONAL_HEADERS ...]]
                        Additional column names for COPY INTO statement (must
                        be used together with --additional_content, the
                        argument quantity must match for both arguments, and
                        value(s) must each be enclosed in single quotes. i.e.
                        --additional_headers my_col_a my_col_g
                        --additional_content 'value_a' 'value_b'). Not
                        applicable to JSON file type or pre-compressed file.
                        [default: Empty List]
  --additional_content [ADDITIONAL_CONTENT [ADDITIONAL_CONTENT ...]]
                        Additional values for COPY INTO statement (must be
                        used together with --additional_headers, the argument
                        quantity must match for both arguments, and value(s)
                        must each be enclosed in single quotes. i.e.
                        --additional_headers my_col_a my_col_g
                        --additional_content 'value_a' 'value_b'). Not
                        applicable to JSON file type or pre-compressed file.
                        [default: Empty List]

optional flags (do not specify value(s)):
  --no_purge            Does not purge files from stage if there is a
                        successful load.
  --remove_split_dir    Removes split directory after load attempt. Not
                        applicable to JSON file type.
  --log                 Write output to log file in addition to console
                        output. Use the --quiet flag not to output to
                        console.
  --log_to_error_table  Log error(s) to an error table. Default table name is
                        ERROR_TABLE or can be set by --error_table argument.
                        Will automatically try to create error log table if
                        doesn't exist.
  --quiet               Prevents output to console.
  --synchronous         Prevents SQL statements from being executed
                        asynchronously.
  --split_file          Enables splitting of source files. Not applicable to
                        JSON file type or already compressed files.
  --use_upload_dir      Enables uploading directory rather than individual
                        files.
  --no_error_on_empty   Return an errorlevel / exit code of 0 instead of 1
                        when file is empty.
  --sf_no_warehouse_resume
                        Do not explicitly resume warehouse if suspended
                        before executing actions.
  --sf_no_compress      Do not compress uncompressed file(s) before
                        uploading.
  --show_timing         Executes requested actions but won't output anything
                        to console except for start, end, and duration of
                        execution timing details