Skip to content

Troubleshooting Guide

This guide helps you diagnose and resolve common issues with Scyllaridae microservices.

General Debugging Steps

1. Check Service Health

First, verify the service is running and responding:

# Check health endpoint
curl -f http://localhost:8080/healthcheck

# Check service logs
docker logs your-microservice-container
kubectl logs deployment/your-microservice

2. Verify Configuration

Ensure your configuration is valid and properly loaded:

# Check configuration file syntax
docker exec your-container cat /app/scyllaridae.yml

# Validate YAML syntax
python -c "import yaml; yaml.safe_load(open('scyllaridae.yml'))"

3. Test with Simple Request

Test with a basic request to isolate the issue:

# Simple POST request
echo "test" | curl -X POST \
  -H "Content-Type: text/plain" \
  --data-binary @- \
  http://localhost:8080/

Logging and Monitoring

Enable Debug Logging

Set the log level to DEBUG for detailed information:

# Docker
docker run -e SCYLLARIDAE_LOG_LEVEL=DEBUG your-image

# Kubernetes
kubectl set env deployment/your-microservice SCYLLARIDAE_LOG_LEVEL=DEBUG

Structured Logging

Scyllaridae uses structured logging. Key log fields include:

  • level: Log level (DEBUG, INFO, ERROR)
  • msg: Log message
  • path: Request path
  • status: HTTP status code
  • duration: Request duration
  • command: Executed command
  • msgId: Message ID from events

Common Issues and Solutions

Authentication Issues

JWT Token Invalid

Symptoms:

  • HTTP 401 Unauthorized responses
  • Error message: "JWT verification failed"

Causes:

  • Expired JWT token
  • Invalid token signature
  • Incorrect JWKS URI
  • Network connectivity to JWKS endpoint

Solutions:

  1. Verify token validity:
  # Decode JWT to check expiration (install jq first)
  echo "your-jwt-token" | cut -d. -f2 | base64 -d | jq .exp
  1. Test JWKS connectivity:
  curl https://your-domain.com/oauth/discovery/keys
  1. Check configuration:
# Ensure correct JWKS URI in scyllaridae.yml
jwksUri: "https://your-domain.com/oauth/discovery/keys"
  1. Temporarily disable JWT for testing:
# Remove or comment out jwksUri
# jwksUri: ""

Missing Authorization Header

Symptoms:

  • HTTP 400 Bad Request
  • Error message: "Missing Authorization header"

Solutions:

  1. Add Authorization header:
  curl -H "Authorization: Bearer your-jwt-token" ...
  1. Check if JWT is required:
# If no authentication needed, ensure jwksUri is empty
# jwksUri: ""

Configuration Issues

Invalid YAML Syntax

Symptoms:

  • Service fails to start
  • Error message contains "yaml" or "parsing"

Solutions:

  1. Validate YAML syntax:
  python -c "import yaml; print(yaml.safe_load(open('scyllaridae.yml')))"
  1. Check indentation (use spaces, not tabs):
# Correct indentation
cmdByMimeType:
  default:
    cmd: "echo"
    args:
      - "hello"
  1. Quote string values with special characters:
jwksUri: "https://example.com/keys" # Quote URLs

Command Not Found

Symptoms:

  • HTTP 500 Internal Server Error
  • Log message: "executable file not found"

Solutions:

  1. Verify command availability:
  docker exec your-container which your-command
  1. Install missing dependencies in Dockerfile:
  FROM islandora/scyllaridae:main
  RUN apk add --no-cache your-required-package
  1. Use full command paths:
cmdByMimeType:
  default:
    cmd: "/usr/bin/convert" # Full path instead of "convert"

Invalid MIME Type Configuration

Symptoms:

  • HTTP 400 Bad Request
  • Error message: "undefined mimeType to build command"

Solutions:

  1. Check allowed MIME types:
allowedMimeTypes:
  - "*" # Allow all, or specify exact types
  - "image/*" # Allow type families
  1. Add missing MIME types:
allowedMimeTypes:
  - "application/pdf"
  - "image/jpeg"
  - "text/plain"
  1. Check Content-Type header:
  curl -H "Content-Type: image/jpeg" ...  # Must match allowed types

Network and Connectivity Issues

Cannot Fetch Source File

Symptoms:

  • HTTP 424 Failed Dependency
  • Log message: "Error fetching source file contents"

Causes:

  • Source URL unreachable
  • Network connectivity issues
  • SSL/TLS certificate problems
  • Authorization issues with source

Solutions:

  1. Test source URL directly:
  curl -I "https://example.com/source-file.pdf"
  1. Check SSL certificates:
  # Add CA certificate to container
  docker run -v ./ca.pem:/app/ca.pem your-image
  1. Verify network connectivity:
  docker exec your-container ping example.com
  1. Check authorization forwarding:
forwardAuth: true # Forward Authorization header to source

Performance Issues

High Memory Usage

Symptoms:

  • Out of memory errors
  • Service becoming unresponsive
  • Container restarts

Solutions:

  1. Monitor memory usage:
  docker stats your-container
  kubectl top pods
  1. Optimize command usage:
# Use streaming commands when possible
cmdByMimeType:
  default:
    cmd: "convert"
    args: ["-", "output:-"] # Stream input/output
  1. Set memory limits:
# Docker Compose
services:
  microservice:
    deploy:
      resources:
        limits:
          memory: 512M
  1. Process smaller files or implement file size limits.

Slow Processing

Symptoms:

  • Long response times
  • Request timeouts

Solutions:

  1. Profile command performance:
  time your-command < input-file > output-file
  1. Optimize command arguments:
# Example: faster image processing
cmdByMimeType:
  "image/*":
    cmd: "convert"
    args: ["-", "-quality", "80", "-resize", "300x300>", "jpg:-"]
  1. Scale horizontally:
# Kubernetes
spec:
  replicas: 3 # Run multiple instances

File Processing Issues

Empty Output

Symptoms:

  • HTTP 200 response with empty body
  • No error messages in logs

Causes:

  • Command produces no output
  • Command writes to stderr instead of stdout
  • Command requires specific arguments

Solutions:

  1. Test command manually:
  echo "test input" | your-command
  1. Check command output redirection:
  # Ensure command writes to stdout, not files
  your-command < input > /dev/stdout
  1. Debug with verbose logging:
# Set SCYLLARIDAE_LOG_LEVEL=DEBUG
environment:
  SCYLLARIDAE_LOG_LEVEL: DEBUG

Corrupted Output

Symptoms:

  • Garbled or invalid output files
  • File format errors

Causes:

  • Binary/text encoding issues
  • Command error output mixed with file output
  • Incorrect command arguments

Solutions:

  1. Ensure binary mode for binary files:
  curl --data-binary "@file.pdf" ...  # Use --data-binary for binary files
  1. Redirect stderr to prevent contamination:
  #!/bin/bash
  your-command 2>/dev/null  # Redirect stderr
  1. Use wrapper scripts for complex commands:
  #!/bin/bash
  input_file=$(mktemp)
  output_file=$(mktemp)

  cat > "$input_file"
  your-command "$input_file" "$output_file" 2>/dev/null
  cat "$output_file"

  rm "$input_file" "$output_file"

Container and Runtime Issues

Service Won't Start

Symptoms:

  • Container exits immediately
  • "Container failed to start" errors

Solutions:

  1. Check container logs:
  docker logs your-container
  1. Verify base image:
  FROM islandora/scyllaridae:main  # Ensure correct base image
  1. Test container interactively:
  docker run -it your-image /bin/bash
  1. Check file permissions:
  # Ensure scyllaridae.yml is readable
  ls -la /app/scyllaridae.yml

Port Binding Issues

Symptoms:

  • "Port already in use" errors
  • Cannot connect to service

Solutions:

  1. Check port availability:
  netstat -tlnp | grep :8080
  lsof -i :8080
  1. Use different port:
  docker run -p 8081:8080 your-image
  1. Check firewall settings:
  # Ensure port is open
  ufw status
  iptables -L