Deploying Serverless Functions

Deploy and configure serverless functions using Nuclio for CVAT

Prerequisites

Before deploying serverless functions, ensure you have:

• CVAT self-hosted installation running
• Docker installed and running
• Nuclio CLI (nuctl) installed
• Access to the CVAT serverless functions repository
• Sufficient system resources (4GB+ RAM recommended for model loading)

Installation

Install Nuclio CLI

curl -s https://api.github.com/repos/nuclio/nuclio/releases/latest \
  | grep -i "browser_download_url.*nuctl.*$(uname)" \
  | cut -d : -f 2,3 \
  | tr -d \" \
  | wget -O nuctl -qi - && chmod +x nuctl

sudo mv nuctl /usr/local/bin/

Clone Serverless Functions

The serverless functions are included in the CVAT repository:

git clone https://github.com/cvat-ai/cvat.git
cd cvat/serverless

Deployment Options

Option 1: Deploy All CPU Functions

Deploy all available functions optimized for CPU:

./deploy_cpu.sh

This script:

1. Builds the OpenVINO base image
2. Creates a cvat project in Nuclio
3. Discovers all function.yaml files
4. Builds Docker images for each function
5. Deploys functions to the local Nuclio platform

Option 2: Deploy All GPU Functions

Deploy GPU-optimized versions (requires NVIDIA GPU and drivers):

./deploy_gpu.sh

This uses function-gpu.yaml files instead of standard function.yaml files.

Option 3: Deploy Individual Functions

Deploy specific functions manually:

# Create Nuclio project
nuctl create project cvat --platform local

# Build base image (for OpenVINO functions)
docker build -t cvat.openvino.base ./openvino/base

# Deploy a specific function
nuctl deploy --project-name cvat \
  --path ./pytorch/facebookresearch/sam/nuclio \
  --file ./pytorch/facebookresearch/sam/nuclio/function.yaml \
  --platform local

Configuration

Function YAML Structure

Each function is defined by a function.yaml file:

metadata:
  name: pth-facebookresearch-sam-vit-h
  namespace: cvat
  annotations:
    name: Segment Anything
    version: 2
    type: interactor
    spec: |
      [
        { "name": "object", "type": "mask" }
      ]
    min_pos_points: 0
    min_neg_points: 0
    startswith_box_optional: true

spec:
  description: Interactive object segmentation with Segment-Anything
  runtime: 'python:3.10'
  handler: main:handler
  eventTimeout: 30s
  
  build:
    image: cvat.pth.facebookresearch.sam.vit_h
    baseImage: ubuntu:22.04
    directives:
      preCopy:
        - kind: RUN
          value: apt-get update && apt-get install -y python3-pip
        - kind: RUN
          value: pip install torch torchvision segment-anything

  triggers:
    myHttpTrigger:
      numWorkers: 2
      kind: 'http'
      attributes:
        maxRequestBodySize: 33554432 # 32MB

Key Configuration Parameters

Metadata Annotations

• name: Display name in CVAT UI
• type: Function type (detector, interactor, tracker, reid)
• spec: Label schema with supported output types
• version: Function version for compatibility

Spec Settings

• runtime: Python version for the function
• handler: Entry point (typically main:handler)
• eventTimeout: Maximum execution time per request
• numWorkers: Number of concurrent workers
• maxRequestBodySize: Maximum request payload size

Environment Variables

Configure CVAT connection when deploying:

nuctl deploy --project-name cvat \
  --path ./function/path \
  --file ./function/path/function.yaml \
  --platform local \
  --env CVAT_FUNCTIONS_REDIS_HOST=cvat_redis_ondisk \
  --env CVAT_FUNCTIONS_REDIS_PORT=6666 \
  --platform-config '{"attributes": {"network": "cvat_cvat"}}'

Connecting to CVAT

Configure Nuclio Settings

In your CVAT deployment, configure Nuclio connection in docker-compose.yml:

services:
  cvat_server:
    environment:
      CVAT_SERVERLESS: 1
      NUCLIO_SCHEME: http
      NUCLIO_HOST: nuclio
      NUCLIO_PORT: 8070
      NUCLIO_FUNCTION_NAMESPACE: nuclio
      NUCLIO_DEFAULT_TIMEOUT: 30
      NUCLIO_INVOKE_METHOD: direct  # or 'dashboard'

Invoke Methods

Direct Invocation (default):

• Calls functions directly via HTTP port
• Lower latency, fewer network hops
• Requires functions accessible from CVAT container

Dashboard Invocation:

• Routes through Nuclio dashboard API
• Better for complex networking scenarios
• Slightly higher latency

Network Configuration

Ensure CVAT and Nuclio containers share a network:

docker network create cvat_cvat

# When deploying functions, specify network:
nuctl deploy --project-name cvat \
  --platform-config '{"attributes": {"network": "cvat_cvat"}}' \
  ...

Verification

List Deployed Functions

nuctl get function --platform local

Expected output:

NAMESPACE | NAME                              | PROJECT | STATE | REPLICAS
nuclio    | pth-facebookresearch-sam-vit-h    | cvat    | ready | 2/2
nuclio    | onnx-wongkinyiu-yolov7            | cvat    | ready | 2/2
...

Test Function Invocation

Test a function directly:

echo '{"image": "base64_encoded_image_data"}' | \
  nuctl invoke pth-facebookresearch-sam-vit-h --method POST

Check CVAT Integration

1. Open CVAT UI
2. Create or open a task
3. Navigate to annotation view
4. Check “Tools” menu for available AI models
5. Verify functions appear under “Magic” or “Automatic annotation”

Troubleshooting

Function Not Appearing in CVAT

Check Nuclio function status:

nuctl get function --platform local

Verify CVAT environment variables:

docker exec cvat_server env | grep NUCLIO

Check CVAT logs:

docker logs cvat_server | grep lambda

Function Build Failures

Increase Docker memory:

• Some models (especially Mask R-CNN) require 4GB+ RAM
• Adjust Docker Desktop settings or system resources

Check base image:

docker images | grep cvat.openvino.base

Rebuild base image:

docker build -t cvat.openvino.base ./openvino/base

Function Timeout Issues

Increase timeout in function.yaml:

spec:
  eventTimeout: 60s  # Increase from 30s

Update CVAT timeout:

environment:
  NUCLIO_DEFAULT_TIMEOUT: 60

Network Connection Errors

Verify network:

docker network inspect cvat_cvat

Check function accessibility:

docker exec cvat_server curl http://nuclio:8070/api/functions

Update invoke method:

environment:
  NUCLIO_INVOKE_METHOD: dashboard  # Try alternative method

Resource Management

Memory Requirements

Typical memory usage per function:

• YOLO v7 (ONNX): ~500MB
• Mask R-CNN (OpenVINO): ~2GB
• SAM (PyTorch): ~2.5GB
• Detectron2 models: ~1-3GB

GPU Support

For GPU acceleration:

1. Install NVIDIA Container Toolkit:

sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

2. Deploy GPU functions:

./deploy_gpu.sh

3. Verify GPU access:

docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

Updating Functions

Update Existing Function

# Delete old function
nuctl delete function pth-facebookresearch-sam-vit-h --platform local

# Redeploy with updated configuration
nuctl deploy --project-name cvat \
  --path ./pytorch/facebookresearch/sam/nuclio \
  --file ./pytorch/facebookresearch/sam/nuclio/function.yaml \
  --platform local

Update All Functions

# Remove all functions
nuctl delete project cvat --platform local

# Redeploy
./deploy_cpu.sh

Best Practices

1. Start Small: Deploy only the functions you need initially
2. Monitor Resources: Track memory and CPU usage during operation
3. Use CPU for Batch: CPU functions work well for background annotation jobs
4. Use GPU for Interactive: GPU accelerates real-time interactor tools
5. Version Control: Track function.yaml changes for reproducibility
6. Test Locally: Verify functions work before deploying to production
7. Network Isolation: Use dedicated networks for security
8. Regular Updates: Keep base images and dependencies updated

Next Steps

Custom Models

Create your own serverless functions

Overview

Learn more about serverless function types