FlowObjects Data Loader
Command Line Parameters
Purpose
The FlowObjects Data Loader is a command line interface (CLI) tool that is designed to efficiently load data to Snowflake. Some of the features include:
- reduce upload time by leveraging compression
- break up large files into smaller optimal chunks
- produce logs to console, file, or a table to audit a data load
- control auto commit
- add column(s) and value(s), i.e. meta-data - - inline, without modifying the source
- easily embed in your existing processes (i.e. automation scripts)
See also: Using CLI switches or environment variables
Data Loader 3.0.0, CLI switches and flags usage
usage: flowobjects_data_loader [-h] [-v] [-s] --file FILE --sf_account
SF_ACCOUNT --sf_username SF_USERNAME
--sf_password SF_PASSWORD --sf_database
SF_DATABASE --sf_warehouse SF_WAREHOUSE
--sf_role SF_ROLE --sf_schema SF_SCHEMA
--sf_table SF_TABLE --sf_file_format
SF_FILE_FORMAT --sf_stage_type SF_STAGE_TYPE
[--sf_autocommit SF_AUTOCOMMIT]
[--sf_stage_name SF_STAGE_NAME]
[--encoding ENCODING] [--delimiter DELIMITER]
[--header_rows HEADER_ROWS]
[--on_error ON_ERROR]
[--transaction_name TRANSACTION_NAME]
[--error_table ERROR_TABLE]
[--split_file_size SPLIT_FILE_SIZE]
[--transformation_file TRANSFORMATION_FILE]
[--copy_into COPY_INTO]
[--additional_headers [ADDITIONAL_HEADERS [ADDITIONAL_HEADERS ...]]]
[--additional_content [ADDITIONAL_CONTENT [ADDITIONAL_CONTENT ...]]]
[--no_purge] [--remove_split_dir] [--log]
[--log_to_error_table] [--quiet]
[--synchronous] [--split_file]
[--use_upload_dir] [--no_error_on_empty]
[--sf_no_warehouse_resume] [--sf_no_compress]
[--show_timing]
The arguments below are divided into switches for which a value must be
provided and flags for which no value should be provided. Most switches can
be used as environment variables instead. Some switches are required while
others are optional. Switches must be lowercase and are shown below with --
in front of them. Environment variable equivalents are shown next to the
switch in all uppercase. For more details, visit:
https://community.flowobjects.com/t/using-cli-switches-or-environment-
variables
arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-s, --sysinfo
required switches (requires value(s) for switch):
--file FILE File(s) or Directory/Directories to Upload (i.e.
file: mydata.csv || files: *.csv || directory of
files: myfilesdir/). If provided file not already
compressed, will automatically compress (in GZIP
format) before uploading (see --sf_no_compress flag
not to automatically compress). Already compressed
supported file formats will pass through without
being compressed again.
--sf_account SF_ACCOUNT
Snowflake account name.
--sf_username SF_USERNAME
Snowflake user name.
--sf_password SF_PASSWORD
Snowflake password.
--sf_database SF_DATABASE
Snowflake database.
--sf_warehouse SF_WAREHOUSE
Snowflake warehouse.
--sf_role SF_ROLE Snowflake role.
--sf_schema SF_SCHEMA
Snowflake schema.
--sf_table SF_TABLE Snowflake table to place file data in (i.e.
DATABASE.SCHEMA.TABLENAME).
--sf_file_format SF_FILE_FORMAT
File format of files to upload (i.e.
EXCEL_DEMO_FORMAT).
--sf_stage_type SF_STAGE_TYPE
Stage type (i.e. TABLE, NAMED, USER).
optional switches (requires value(s) for switch):
--sf_autocommit SF_AUTOCOMMIT
Control whether on connection close all changes are
automatically committed [default: True]. Set to True
to enable autocommit or False to disable it.
--sf_stage_name SF_STAGE_NAME
Stage name if using a named or table stage (i.e.
DATABASE.SCHEMA.STAGENAME) [default: None]
--encoding ENCODING Encoding for file to upload [default: None]. If not
specified, read from Snowflake file format.
--delimiter DELIMITER
Line delimiter of file to upload [default: None]. If
not specified, read from Snowflake file format. Not
applicable to JSON file type.
--header_rows HEADER_ROWS
Number of header rows to skip [default: 0]. Not
applicable to JSON file type.
--on_error ON_ERROR Setting for 'COPY INTO' command if an error is
encountered (i.e. CONTINUE, SKIP_FILE,
SKIP_FILE_, SKIP_FILE_, ABORT_STATEMENT).
[default: CONTINUE].
--transaction_name TRANSACTION_NAME
Transaction name [default: FILE_UPLOAD]
--error_table ERROR_TABLE
Table to write errors to (i.e.
DATABASE.SCHEMA.ERRORTABLENAME). If not specified,
will default to ERROR_TABLE and must be usedtogether
with --log_to_error_table
--split_file_size SPLIT_FILE_SIZE
Size of split record files in bytes. [default:
100000000 (100 MB)]. Not applicable to JSON file type
or pre-compressed file.
--transformation_file TRANSFORMATION_FILE
JSON file specifying transformations. [default: None]
--copy_into COPY_INTO
JSON file with custom COPY INTO statement to be used
instead of auto-generated default. [default: None]
--additional_headers [ADDITIONAL_HEADERS [ADDITIONAL_HEADERS ...]]
Additional column names for COPY INTO statement (must
be used together with --additional_content, the
argument quantity must match for both arguments, and
value(s) must each be enclosed in single quotes. i.e.
--additional_headers my_col_a my_col_g
--additional_content 'value_a' 'value_b'). Not
applicable to JSON file type or pre-compressed file.
[default: Empty List]
--additional_content [ADDITIONAL_CONTENT [ADDITIONAL_CONTENT ...]]
Additional values for COPY INTO statement (must be
used together with --additional_headers, the argument
quantity must match for both arguments, and value(s)
must each be enclosed in single quotes. i.e.
--additional_headers my_col_a my_col_g
--additional_content 'value_a' 'value_b'). Not
applicable to JSON file type or pre-compressed file.
[default: Empty List]
optional flags (do not specify value(s)):
--no_purge Does not purge files from stage if there is a
successful load.
--remove_split_dir Removes split directory after load attempt. Not
applicable to JSON file type.
--log Write output to log file in addition to console
output. Use the --quiet flag not to output to
console.
--log_to_error_table Log error(s) to an error table. Default table name is
ERROR_TABLE or can be set by --error_table argument.
Will automatically try to create error log table if
doesn't exist.
--quiet Prevents output to console.
--synchronous Prevents SQL statements from being executed
asynchronously.
--split_file Enables splitting of source files. Not applicable to
JSON file type or already compressed files.
--use_upload_dir Enables uploading directory rather than individual
files.
--no_error_on_empty Return an errorlevel / exit code of 0 instead of 1
when file is empty.
--sf_no_warehouse_resume
Do not explicitly resume warehouse if suspended
before executing actions.
--sf_no_compress Do not compress uncompressed file(s) before
uploading.
--show_timing Executes requested actions but won't output anything
to console except for start, end, and duration of
execution timing details