WSO2 Plugin

The Treblle WSO2 SDK integrates with WSO2’s Global Synapse Handler to capture API request/response data and send it asynchronously to Treblle’s monitoring service, ensuring zero impact on API performance.

Supported Versions

Key Features

• Non-blocking: Never blocks or slows down API requests, even on failures - handler always returns true
• Thread-safe: Uses concurrent queues and atomic operations
• Load-balanced: Distributes traffic across multiple Treblle endpoints
• Sensitive data masking: Automatically redacts passwords, tokens, credit cards, SSN, etc.
• Multi-tenant support: Filter by tenant domains
• Route path capture: Captures API resource template patterns (e.g., /users/{userId})
• API identification: Captures API UUID and name for accurate tracking
• Content-type aware: Only captures application/json request/response bodies
• CORS-aware: Automatically filters OPTIONS requests

Prerequisites

Before you begin, ensure you have:

• Java: Version 8 or higher (java -version)
• Maven: Version 3.6.0 or later (mvn -version)
• WSO2 API Manager: Version 3.2.x or later
• Treblle Account: With SDK token and API key (Get started )

Getting Started with WSO2 API Manager

Step 1: Install WSO2 API Manager

1. Download WSO2 API Manager from the official WSO2 website .

2. Unzip the archive:

   unzip wso2am-4.3.0.zip
   cd wso2am-4.3.0/bin/
   ./api-manager.sh

3. Open the Admin Console in your browser:

   • URL: https://localhost:9443/carbon
   • Username: admin
   • Password: admin

Step 2: Create an API in WSO2 API Manager

1. Log in to the Publisher Portal at https://localhost:9443/publisher.

2. Use the default credentials:

   • Username: admin
   • Password: admin

3. Create a New API:

   • Choose the API type (e.g., REST API, SOAP API)
   • Provide details like API Name, Context Path, Version, and Endpoint URL
   
   • Publish the API
   

4. Test the API:

   • Log in to the Developer Portal at https://localhost:9443/devportal/apis
   

5. Subscribe to the API:

   • Go to the Subscription section
   
   • Use the Subscription and Key Generation Wizard to generate an access token
   

Step 3: Test the API

• Return to the Developer Portal and locate the API you subscribed to

• Click the Try Out option to test your API directly within the portal

  

• Select the key type for testing:

  • Sandbox: For testing the API in a non-production environment
  • Production: For live use in production
  

• Choose the method you want to test (e.g., GET)

  

• Execute the request by providing the required parameters

  

Step 4: Set Up Treblle Account

Follow these steps to create your Treblle account and get API credentials:

1. Visit the Treblle website  and log in or sign up  for an account if you don’t have one.

2. Once logged in, click “New API” and provide the following details:

   • API Name (e.g., WSO2-4.3.0)
   • Environment (Production, Staging, Development, etc.)
   • Category (Healthcare, Finance, E-commerce, etc.)
   • Type (Internal, External, Partner)
   • Tags (Optional - for organizing your APIs)
   

3. Select WSO2 from the Integrations page to access WSO2-specific setup instructions:

   

   The setup wizard will provide:

   • Step-by-step installation commands
   • Configuration snippets for deployment.toml
   • Logger configuration for log4j2.properties
   • Your unique API Key and SDK Token

4. After completing the setup, Treblle will generate your credentials:

   • API Key - Project identifier
   • SDK Token - Authentication token

Installation

Step 1: Build from Source

Building for Default Version (WSO2 APIM 4.3.x):

# Clone the repository
git clone https://github.com/Treblle/treblle-wso2.git
cd treblle-wso2
 
# Build for default version (4.3.x)
mvn clean install

The JAR artifact will be created in the target/ directory as treblle-data-publisher-4.3.x-1.0.0.jar

Building for Specific WSO2 Versions:

# For WSO2 APIM 3.2.x
mvn clean install -P wso2am-3.2
 
# For WSO2 APIM 4.0.x
mvn clean install -P wso2am-4.0
 
# For WSO2 APIM 4.1.x
mvn clean install -P wso2am-4.1
 
# For WSO2 APIM 4.2.x
mvn clean install -P wso2am-4.2
 
# For WSO2 APIM 4.3.x (default)
mvn clean install -P wso2am-4.3
 
# For WSO2 APIM 4.4.x
mvn clean install -P wso2am-4.4

Step 2: Deploy the JAR

Copy the built JAR artifact that matches your WSO2 version to your API Manager gateway:

# Example for WSO2 APIM 4.3.x (default)
cp target/treblle-data-publisher-4.3.x-1.0.0.jar <APIM_HOME>/repository/components/lib/
 
# For other versions, use the appropriate JAR file:
# cp target/treblle-data-publisher-3.2.x-1.0.0.jar <APIM_HOME>/repository/components/lib/
# cp target/treblle-data-publisher-4.0.x-1.0.0.jar <APIM_HOME>/repository/components/lib/
# cp target/treblle-data-publisher-4.4.x-1.0.0.jar <APIM_HOME>/repository/components/lib/

Replace <APIM_HOME> with your WSO2 API Manager installation directory (e.g., /opt/wso2am-4.3.0).

Step 3: Configure the Global Synapse Handler

Add the following configuration at the beginning of <APIM_HOME>/repository/conf/deployment.toml:

[synapse_handlers.treblle_publisher]
enabled=true
class="com.treblle.wso2publisher.handlers.APILogHandler"

Step 4: Configure Logging

Add the following to <APIM_HOME>/repository/conf/log4j2.properties:

# Add treblle_publisher to the loggers list (find the existing loggers line and append)
loggers = treblle_publisher, AUDIT_LOG, ...
 
# Add these lines at the end of the file
logger.treblle_publisher.name = com.treblle.wso2publisher
logger.treblle_publisher.level = INFO
logger.treblle_publisher.appenderRef.CARBON_LOGFILE.ref = CARBON_LOGFILE

Step 5: Set Environment Variables

Before starting the WSO2 API Manager, configure the required environment variables.

For macOS/Linux:

# Required variables
export TREBLLE_SDK_TOKEN="your-sdk-token"      # Your Treblle SDK Token
export TREBLLE_API_KEY="your-api-key"          # Your Treblle API key
export TREBLLE_GATEWAY_URL="https://your-gateway-url"  # Your WSO2 gateway URL
 
# Optional variables with defaults
export TREBLLE_QUEUE_SIZE=20000                 # Max size of the request queue (default: 20000)
export TREBLLE_WORKER_THREADS=1                 # Number of worker threads (default: 1)
export ADDITIONAL_MASK_KEYWORDS="Authorization,X-API-Key,token"  # Additional fields to mask
export TREBLLE_ENABLED_TENANT_DOMAINS="carbon.super"  # Enabled tenant domains (default: all)

For Windows:

:: Required variables
set TREBLLE_SDK_TOKEN=your-sdk-token
set TREBLLE_API_KEY=your-api-key
set TREBLLE_GATEWAY_URL=https://your-wso2-gateway-url
 
:: Optional variables
set TREBLLE_QUEUE_SIZE=20000
set TREBLLE_WORKER_THREADS=1
set ADDITIONAL_MASK_KEYWORDS=Authorization,X-API-Key,token
set TREBLLE_ENABLED_TENANT_DOMAINS=carbon.super

Verifying Environment Variables:

To check that your environment variables are properly set:

# For Linux/macOS
echo $TREBLLE_SDK_TOKEN
echo $TREBLLE_API_KEY
echo $TREBLLE_GATEWAY_URL
 
# For Windows
echo %TREBLLE_SDK_TOKEN%
echo %TREBLLE_API_KEY%
echo %TREBLLE_GATEWAY_URL%

Step 6: Generate OAuth2 Token for Testing

To test your API with Treblle monitoring:

• Log in to the Developer Portal at https://localhost:9443/devportal

• Under the Subscription section, click on your created application

  

• Select either Production or Sandbox as the key type

  

• Click Generate Key for the chosen environment and copy the generated access token

• This token is essential for accessing protected resources in your API

Step 7: Start WSO2 API Manager

Start your WSO2 API Manager gateway:

cd <APIM_HOME>/bin
./api-manager.sh  # or api-manager.bat on Windows

Check the logs to verify the extension loaded successfully:

# View all WSO2 logs
tail -f <APIM_HOME>/repository/logs/wso2carbon.log
 
# Filter for Treblle-specific logs
tail -f <APIM_HOME>/repository/logs/wso2carbon.log | grep -i treblle

You should see Treblle-related log entries indicating successful initialization.

Step 8: Verify Treblle Integration

1. Check WSO2 logs for Treblle activity:

   tail -f /path/to/your/wso2am-4.3.0/repository/logs/wso2carbon.log | grep -i treblle

2. Look for success messages (not 403 Forbidden errors):

   • Good: Event successfully published.
   • Bad: Event publishing failed for sdk token: your-sdk-token with status code: 403

3. Check your Treblle dashboard to see captured API requests.

Configuration Reference

Default Masked Keywords

The following keywords are automatically masked in headers and request/response bodies:

• password, pwd, secret, password_confirmation
• cc, card_number, ccv
• ssn, credit_score

Use ADDITIONAL_MASK_KEYWORDS to add more sensitive fields (e.g., Authorization,X-API-Key,token).

Treblle Endpoint Configuration

Load Balancing (Default Behavior):

If TREBLLE_GATEWAY_URL environment variable is NOT set, the extension automatically load-balances requests across Treblle’s endpoints using round-robin:

• https://rocknrolla.treblle.com
• https://punisher.treblle.com
• https://sicario.treblle.com

Custom Endpoint:

If TREBLLE_GATEWAY_URL is set in environment variables, all data is sent to your specified endpoint:

export TREBLLE_GATEWAY_URL="https://custom-treblle.your-domain.com"

Use cases for custom endpoints:

• On-premise Treblle installations
• Custom routing or proxy configurations
• Regional endpoints for compliance
• Development/testing environments

Global vs API-Level Handlers

Treblle uses the Global Synapse Handler approach which means:

• All APIs are automatically monitored without individual configuration
• New APIs are automatically included when published
• Consistent monitoring across your entire API infrastructure
• Easier management and maintenance
• Single configuration point in deployment.toml

What Metadata Does Treblle Capture?

Treblle automatically captures comprehensive metadata for each API request:

Request Data:

• Timestamp
• Source IP address (checks X-FORWARDED-FOR header first)
• HTTP method (GET, POST, PUT, DELETE, etc.)
• Request URL and path
• Request headers
• Request body (JSON only)
• User agent
• Route path template (e.g., /users/{userId}/posts)

Response Data:

• HTTP status code
• Response headers
• Response body (JSON only)
• Response size (in bytes)
• Load time (response time in microseconds)

API Identification:

• API name (internal_name)
• API UUID (internal_id)

Server Information:

• Server IP address
• Timezone
• Operating system details (name, version, architecture)
• Software version
• Protocol and encoding

Error Tracking:

• Error type and messages for 4xx/5xx responses
• Error source and details

Multi-Tenant Support

For WSO2 deployments with multiple tenants:

Configure Tenant Domains

export TREBLLE_ENABLED_TENANT_DOMAINS="carbon.super,tenant1.com,tenant2.com"

Tenant-Specific Monitoring

• Treblle tracks which tenant each API request belongs to
• Dashboard shows requests grouped by tenant domain
• Supports both super tenant (carbon.super) and sub-tenants
• Only monitors APIs from enabled tenant domains

Advanced Configuration

High-Volume Deployments

For production environments with high API traffic:

# Increase queue size for high-volume scenarios
export TREBLLE_QUEUE_SIZE=50000
 
# Add more worker threads for parallel processing
export TREBLLE_WORKER_THREADS=3

Performance Tuning Tips:

• Start with default settings (queue size: 20,000, worker threads: 1)
• Monitor queue full errors in logs
• Increase TREBLLE_QUEUE_SIZE if you see “Event queue is full” messages
• Increase TREBLLE_WORKER_THREADS for better throughput (2-4 threads recommended)
• Each worker thread uses the pooled HTTP client for efficient connection reuse

Custom Masking Configuration

Extend the default masked fields with your sensitive data:

# Mask additional fields beyond the defaults
export ADDITIONAL_MASK_KEYWORDS="Authorization,X-API-Key,Custom-Auth-Header,Bearer-Token,Session-ID"

Masking Behavior:

• Case-insensitive matching (e.g., “password” matches “Password”, “PASSWORD”)
• Applies to both request and response headers and bodies
• Nested JSON objects are recursively masked
• Masked value is replaced with "****"

Connection Pooling

The extension uses a pooled HTTP client for optimal performance:

• Max Total Connections: 100
• Max Connections Per Route: 20
• Connection Timeout: 5 seconds
• Socket Timeout: 10 seconds
• Connection Request Timeout: 5 seconds
• Validate After Inactivity: 2 seconds

These settings are optimized for production use and don’t require configuration.

Troubleshooting

Extension Not Loading

Check logs in <APIM_HOME>/repository/logs/wso2carbon.log:

grep -i treblle wso2carbon.log

Verify JAR location:

ls -la <APIM_HOME>/repository/components/lib/treblle-data-publisher-*.jar
 
# Ensure the version matches your WSO2 APIM version
# Example output: treblle-data-publisher-4.3.x-1.0.0.jar

Verify deployment.toml configuration is at the beginning of the file and properly formatted.

No Data in Treblle Dashboard

1. Verify environment variables are set:

   echo $TREBLLE_SDK_TOKEN
   echo $TREBLLE_API_KEY
   echo $TREBLLE_GATEWAY_URL

2. Check for errors in logs:

   grep -i "treblle.*error" wso2carbon.log
   tail -f wso2carbon.log | grep "treblle\|PublisherClient\|APILogHandler"

3. Verify tenant domain filtering (if using TREBLLE_ENABLED_TENANT_DOMAINS):

   • Ensure your API’s tenant domain is in the allowed list
   • Default tenant is usually carbon.super
   • Check logs for “Tenant domain not enabled” messages

4. Check content type - Only application/json requests/responses are captured

   • Verify your API uses Content-Type: application/json
   • Other content types are intentionally skipped

5. Verify OPTIONS requests are not expected - CORS preflight requests are automatically filtered

Queue Full Errors

If you see “Event queue is full” errors in logs:

# Error message example:
# Event queue is full. Starting to drop events.
# Event queue is full. Events dropped so far - 1000

Solutions:

1. Increase queue size:

   export TREBLLE_QUEUE_SIZE=50000

2. Add more worker threads:

   export TREBLLE_WORKER_THREADS=2  # or 3

3. Check network connectivity to Treblle endpoints:

   # Test connectivity to default endpoints
   curl -I https://rocknrolla.treblle.com
   curl -I https://punisher.treblle.com
   curl -I https://sicario.treblle.com

Publishing Failures

Check for HTTP status codes in logs:

grep -i "Event publishing failed" wso2carbon.log

Common status codes:

• 200-204: Success - Event successfully published
• 400-499: Client error - Check your TREBLLE_SDK_TOKEN and TREBLLE_API_KEY
• 500-599: Server error - Treblle service may be temporarily unavailable

If you see 403 Forbidden:

• Verify your SDK token is correct
• Verify your API key is correct
• Check that you’re not using placeholder text like “your-sdk-token”

Memory Issues

If experiencing memory pressure:

1. Reduce TREBLLE_QUEUE_SIZE (default 20,000)

   export TREBLLE_QUEUE_SIZE=10000

2. Reduce TREBLLE_WORKER_THREADS (default 1)

   export TREBLLE_WORKER_THREADS=1

3. Use tenant domain filtering to monitor specific APIs:

   export TREBLLE_ENABLED_TENANT_DOMAINS="carbon.super"

Handler Not Executing

Verify configuration:

1. Check deployment.toml has the correct configuration at the beginning
2. Verify environment variables are set before starting WSO2
3. Check for startup errors in logs:

   grep -i "error\|exception" wso2carbon.log | grep -i treblle

Enable debug logging:

In log4j2.properties, change the log level to DEBUG:

logger.treblle_publisher.level = DEBUG

Restart WSO2 and check logs for detailed debugging information:

tail -f wso2carbon.log | grep -i treblle

Database Lock Errors

If you see database lock errors on startup:

# Kill any running WSO2 processes
pkill -f "org.wso2.carbon.bootstrap.Bootstrap"
 
# Remove lock files
cd <APIM_HOME>/repository/database/
rm -f *.lock *.lck *.tmp

SSL Certificate Issues

For development/testing:

• Use the -k flag with cURL for testing: curl -k https://...
• In API Testing tools, disable SSL certificate verification in settings
• For development, you can use HTTP port 8280 instead of HTTPS 8243

Testing the Integration

Step 1: Create and Publish an API

Follow the WSO2 documentation to create and publish a test API in the Publisher Portal at https://localhost:9443/publisher.

Step 2: Subscribe and Generate Token

1. Log in to the Developer Portal at https://localhost:9443/devportal
2. Subscribe to your test API
3. Generate an access token (Production or Sandbox)

Step 3: Test API Calls

GET Request:

curl -k -X GET \
  "https://localhost:8243/test/v1/users" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

POST Request:

curl -k -X POST \
  "https://localhost:8243/test/v1/users" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"email": "john.doe@example.com", "first_name": "John", "last_name": "Doe"}'

PUT Request:

curl -k -X PUT \
  "https://localhost:8243/test/v1/users/123" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"email": "john.doe@example.com", "first_name": "John", "last_name": "Smith"}'

DELETE Request:

curl -k -X DELETE \
  "https://localhost:8243/test/v1/users/123" \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

Step 4: Verify in Treblle Dashboard

1. Log in to your Treblle dashboard at https://platform.treblle.com 
2. Navigate to your Workspace -> API -> Request Tab

Getting Help

If you continue to experience issues:

1. Enable debug logging in log4j2.properties:

   logger.treblle_publisher.level = DEBUG

2. Check console output for detailed error messages

3. Verify credentials are correct in Treblle dashboard

4. Test with a simple endpoint first before complex APIs