Models Management

The Models Management section provides guidance on how to manage and configure models for analytics and semantic analysis tasks within Flock. These tasks involve processing and analyzing text, embeddings, and other data types using pre-configured models, either system-defined or user-defined, based on specific use cases. Each database is configured with its own model management table during the initial load.

• 1. Model Configuration
  • max_batch_size
  • batch_size (deprecated)
  • is_async
  • rate_limit
  • usage_limit
• 2. Management Commands
• 3. Global and Local Models
  • Create Models
• 4. SQL Query Examples
  • Semantic Text Completion
  • Semantic Search

1. Model Configuration

Models are stored in a table with the following structure:

Column Name	Description
Model Name	Unique identifier for the model
Model Type	Specific model type (e.g., gpt-4, llama3)
Provider	Source of the model (e.g., openai, azure, ollama)
Model Arguments	JSON configuration parameters. For user-defined models: only tuple_format, max_batch_size, batch_size (deprecated), model_parameters, is_async, rate_limit, and usage_limit are allowed. tuple_format can be one of: JSON, XML, or Markdown. max_batch_size must be greater than 0 and controls the maximum number of tuples sent in a single provider request. model_parameters is a JSON object of provider-specific settings. is_async is a boolean (default true) that controls whether scalar functions batch completion requests in parallel before collecting responses. rate_limit is an optional positive integer for maximum provider requests per minute, scoped per Flock model_name. usage_limit is an optional JSON object for cumulative token quotas, also scoped per Flock model_name.

max_batch_size

max_batch_size sets the maximum number of input tuples Flock groups into a single provider request. Each request counts as one call against rate_limit and provider quotas.

Default: 16.

CREATE MODEL('batched-gpt4o', 'gpt-4o', 'openai', {"max_batch_size": 64});

You can also override max_batch_size inline when calling a function:

SELECT llm_complete(
    {'model_name': 'gpt-4o', 'max_batch_size': 32},
    {'prompt': 'Summarize', 'context_columns': [{'data': text}]}
) FROM my_table;

Token-limit retries

If a batch exceeds the model context window, Flock automatically retries with smaller batches instead of failing the whole query.

Scalar functions (llm_complete, llm_filter, llm_embedding):

• Flock starts from min(max_batch_size, row_count).
• On a context-window / token-limit error, the failed batch is retried at half the size: 64 → 32 → 16 → …
• After a successful batch, the next chunk starts again at the configured max_batch_size.
• If a batch cannot be reduced further (batch_size of 1 still fails), affected rows return NULL.

Sync (is_async: false) retries batches sequentially. Async (is_async: true, the default) splits only the failed batch and re-queues the smaller pieces; successful batches in the same round are kept.

Aggregate functions (llm_reduce, llm_rerank, llm_first, llm_last):

• Flock shrinks the working batch by 10% on token-limit errors and retries the same rows.
• If the batch size reaches zero, Flock raises an error instead of returning NULL.

This automatic backoff is separate from usage_limit, which tracks cumulative token quotas and fails immediately once exceeded.

batch_size (deprecated)

batch_size is a deprecated alias for max_batch_size. Existing models and inline overrides that use batch_size continue to work, but Flock logs a warning when batch_size is used.

Prefer max_batch_size in new models and queries. If both are set, max_batch_size takes precedence.

-- Deprecated
CREATE MODEL('legacy-model', 'gpt-4o', 'openai', {"batch_size": 16});

-- Preferred
CREATE MODEL('current-model', 'gpt-4o', 'openai', {"max_batch_size": 16});

is_async

is_async applies to llm_complete and llm_filter. When true (the default), all batches are queued in parallel and responses are collected in a single call. When false, batches are processed one at a time synchronously.

-- Default: async batching
CREATE MODEL('async-model', 'gpt-4o', 'openai', {"max_batch_size": 16});

-- Explicit synchronous batching
CREATE MODEL('sync-model', 'gpt-4o', 'openai', {"max_batch_size": 16, "is_async": false});

You can also pass is_async inline when calling a function:

SELECT llm_complete(
    {'model_name': 'gpt-4o', 'is_async': false},
    {'prompt': 'Summarize', 'context_columns': [{'data': text}]}
) FROM my_table;

rate_limit

rate_limit sets the maximum number of provider requests per minute for a Flock model, keyed by model_name. This helps avoid exceeding provider quotas on free or shared API tiers.

When rate_limit is set:

• Flock spaces outgoing provider requests so the configured requests-per-minute cap is not exceeded.
• max_batch_size and rate_limit are independent: max_batch_size controls tuples per request; rate_limit throttles how many provider requests are sent per minute.
• The limit applies across completions, embeddings, and transcriptions for that model.
• Sync and async scalar execution both respect the same per-model limit.

-- Limit to 30 requests per minute
CREATE MODEL('limited-gpt4o', 'gpt-4o', 'openai', {"max_batch_size": 64, "rate_limit": 30});

-- Inline override in a function call
SELECT llm_complete(
    {'model_name': 'gpt-4o', 'rate_limit': 30},
    {'prompt': 'Summarize', 'context_columns': [{'data': text}]}
) FROM my_table;

If rate_limit is omitted, Flock does not apply request-per-minute throttling.

usage_limit

usage_limit sets cumulative token quotas for a Flock model, keyed by model_name. Like rate_limit, it is scoped per model; unlike rate_limit, which throttles how many provider requests are sent per minute, usage_limit tracks provider-reported token usage across calls and fails once a quota is exceeded.

Supported fields (at least one is required):

• prompt_tokens_limit: maximum cumulative input/prompt tokens
• completion_tokens_limit: maximum cumulative output/completion tokens
• total_tokens_limit: maximum cumulative total tokens (prompt + completion)

When a limit is exceeded, Flock raises a usage-limit error. This is separate from context-window overflow errors that trigger automatic max_batch_size retries (see above).

-- Cap total tokens for a model
CREATE MODEL('budget-gpt4o', 'gpt-4o', 'openai', {
    "max_batch_size": 16,
    "usage_limit": {"total_tokens_limit": 50000}
});

-- Inline override in a function call
SELECT llm_complete(
    {
        'model_name': 'gpt-4o',
        'usage_limit': {'prompt_tokens_limit': 10000, 'completion_tokens_limit': 5000}
    },
    {'prompt': 'Summarize', 'context_columns': [{'data': text}]}
) FROM my_table;

If usage_limit is omitted, Flock does not enforce cumulative token quotas.

2. Management Commands

• Retrieve all available models

GET
MODELS;

• Retrieve details of a specific model

GET
MODEL 'model_name';

• Create a new user-defined model

-- User-defined model (only tuple_format, max_batch_size, batch_size, model_parameters, is_async, rate_limit, and usage_limit allowed in JSON)
-- tuple_format can be "JSON", "XML", or "Markdown"
CREATE
MODEL(
    'model_name',
    'model',
    'provider',
    {
        "tuple_format": "JSON",
        "batch_size": 8,
        "model_parameters": {
            "temperature": 0.2,
            "top_p": 0.95
        },
        "is_async": true
    }
);
CREATE
MODEL(
    'model_name',
    'model',
    'provider',
    {
        "tuple_format": "XML",
        "batch_size": 8,
        "model_parameters": {
            "n": 3,
            "frequency_penalty": 0.1
        }
    }
);
CREATE
MODEL(
    'model_name',
    'model',
    'provider',
    {
        "tuple_format": "Markdown",
        "batch_size": 8
    }
);

• Modify an existing user-defined model

-- Update user-defined model (same JSON rules as CREATE)
-- tuple_format can be "JSON", "XML", or "Markdown"
UPDATE MODEL(
    'model_name',
    'model',
    'provider'
);
UPDATE MODEL(
    'model_name',
    'model',
    'provider',
    {
        "tuple_format": "JSON",
        "batch_size": 8,
        "model_parameters": {
            "temperature": 0.2,
            "top_p": 0.95
        },
        "is_async": true
    }
);
UPDATE MODEL(
    'model_name',
    'model',
    'provider',
    {
        "tuple_format": "XML",
        "batch_size": 8,
        "model_parameters": {
            "n": 3,
            "frequency_penalty": 0.1
        }
    }
);
UPDATE MODEL(
    'model_name',
    'model',
    'provider',
    {
    "tuple_format": "Markdown",
    "batch_size": 8
    }
);

• Remove a user-defined model

DELETE
MODEL 'model_name';

3. Global and Local Models

Model creation is database specific. If you want it to be available irrespective of the database, use the GLOBAL keyword. LOCAL is the default if not specified.

Create Models

• Create a global model:

CREATE GLOBAL MODEL
('model_name', 'model_type', 'provider'
);
CREATE GLOBAL MODEL
(
    'model_name', 'model_type', 'provider', {
    "tuple_format"
    :
    "JSON",
    "batch_size": 8,
    "model_parameters": {
            "temperature": 0.2,
    "top_p": 0.95
        }
    }
);
CREATE GLOBAL MODEL
(
    'model_name', 'model_type', 'provider',
    {
        "tuple_format": "XML",
        "batch_size": 8
    }
);
CREATE GLOBAL MODEL
(
    'model_name', 'model_type', 'provider',
    {
        "tuple_format": "Markdown"
    }
);

• Create a local model (default if no type is specified):

CREATE LOCAL MODEL('model_name', 'model_type', 'provider');
CREATE LOCAL MODEL(
    'model_name', 'model_type', 'provider',
    {
        "tuple_format": "JSON",
        "batch_size": 8,
        "model_parameters": {
            "temperature": 0.2
        }
    }
);
CREATE LOCAL MODEL
(
    'model_name', 'model_type', 'provider',
    {
        "tuple_format": "XML",
        "batch_size": 8,
        "model_parameters": {
            "n": 3
        }
    }
);
CREATE LOCAL MODEL
(
    'model_name', 'model_type', 'provider',
    {
        "tuple_format": "Markdown",
        "batch_size": 8,
        "model_parameters": {
            "top_p": 0.95
        }
    }
);
CREATE
MODEL(
    'model_name',
    'model_type',
    'provider'
);
CREATE
MODEL(
    'model_name',
    'model_type',
    'provider',
    {
        "tuple_format": "JSON",
        "batch_size": 8,
        "model_parameters": {
            "temperature": 0.2
        }
    }
);
CREATE
MODEL(
    'model_name',
    'model_type',
    'provider',
    {
        "tuple_format": "XML",
        "batch_size": 8,
        "model_parameters": {
            "n": 3
        }
    }
);
CREATE
MODEL(
    'model_name',
    'model_type',
    'provider',
    {
        "tuple_format": "Markdown",
        "batch_size": 8,
        "model_parameters": {
            "top_p": 0.95
        }
    }
);

• Toggle a model's state between global and local:

UPDATE MODEL 'model_name' TO GLOBAL;
UPDATE MODEL 'model_name' TO LOCAL;

All other queries remain the same for both global and local models.

4. SQL Query Examples

Semantic Text Completion

SELECT llm_complete(
    {'model_name': 'gpt-4o'},
    {'prompt_name': 'product-description'},
    {'input_text': product_description}
) AS generated_description
FROM products;

Semantic Search

SELECT llm_complete(
    {'model_name': 'semantic_search_model'},
    {'prompt_name': 'search-query'},
    {'search_query': query}
) AS search_results
FROM search_data;