Bug: Fireworks CompeleteText endpoint should propagate max_completion…

[ianbbqzy]

…_tokens as well

just like it does for CompleteChat endpoint

Description

Motivation

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)

How Has This Been Tested?

• Unit Tests
• Integration Tests
• Manual Testing

Screenshots (if applicable)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Related Issues

[matter-code-review]

Description

Summary By MatterAI

🔄 What Changed

This Pull Request introduces significant new functionality and broad provider compatibility enhancements. Key changes include the addition of three new default plugins: allowedRequestTypes for controlling API endpoint access, modelRules for enforcing model usage policies, and regexReplace for content redaction and transformation. New API handlers for /v1/images/edits and /v1/messages/count_tokens have been implemented, expanding the gateway's capabilities. Support for new AI providers (Meshy, Tripo3D, NextBit) has been integrated, alongside extensive updates to existing providers (Anthropic, Azure AI, Bedrock, DashScope, DeepSeek, Google Vertex AI, Google, Mistral AI, OpenAI, Together AI) to standardize finish reason transformations, enhance streaming, and improve parameter handling. Dockerfile updates streamline package upgrades, and WebSocket error handling has been made more robust.

🔍 Impact of the Change

These changes substantially extend the Portkey AI gateway's feature set, offering more granular control over API requests, content processing, and model routing. The new plugins provide powerful tools for security, compliance, and custom logic. Expanded provider support and standardized response transformations improve interoperability and developer experience. The introduction of imageEdits and messagesCountTokens endpoints broadens the range of AI tasks supported, while WebSocket and streaming improvements enhance reliability and performance for real-time interactions. The max_completion_tokens bug fix mentioned in the previous summary is implicitly covered by the broader refactoring of provider response transforms.

📁 Total Files Changed

• Dockerfile: Removed redundant apk update command.
• package.json: Updated @hono/node-ws dependency and incremented package version.
• plugins/default/allowedRequestTypes.ts: Added new plugin for request type allow/block listing.
• plugins/default/default.test.ts: Added comprehensive unit tests for allowedRequestTypes plugin and fixed a typo in an import path.
• plugins/default/modelRules.ts: Added new plugin for enforcing model usage rules based on metadata.
• plugins/default/regexReplace.ts: Added new plugin for regex-based content replacement/redaction.
• plugins/index.ts: Integrated new modelRules, regexReplace, and allowedRequestTypes plugins.
• plugins/portkey/gibberish.ts: Refactored fetchPortkey call for consistent destructuring.
• plugins/portkey/language.ts: Refactored fetchPortkey call for consistent destructuring.
• plugins/portkey/moderateContent.ts: Refactored fetchPortkey call for consistent destructuring.
• src/globals.ts: Added new AI providers (Meshy, Tripo3D, NextBit) and new video MIME types.
• src/handlers/handlerUtils.ts: Added URL pathname to ConditionalRouter context and expanded Azure AI configuration options.
• src/handlers/imageEditsHandler.ts: Added new handler for image editing API requests.
• src/handlers/messagesCountTokensHandler.ts: Added new handler for message token counting API requests.
• src/handlers/realtimeHandlerNode.ts: Improved WebSocket connection handling and error logging.
• src/handlers/services/cacheService.ts: Added imageEdit to non-cacheable endpoints.
• src/handlers/services/requestContext.ts: Updated isStreaming logic for imageEdit requests.
• src/handlers/services/responseService.ts: Refined content-encoding header deletion for Node.js runtime.
• src/handlers/streamHandler.ts: Enhanced error logging and added robust writer closing for streaming.
• src/handlers/websocketUtils.ts: Improved WebSocket error handling with dedicated listener and normal close event management.
• src/index.ts: Registered new imageEditsHandler and messagesCountTokensHandler routes.
• src/providers/anthropic/api.ts: Added messagesCountTokens endpoint mapping.
• src/providers/anthropic/chatComplete.ts: Initialized streamState.toolIndex and simplified increment logic.
• src/providers/anthropic/complete.ts: Refactored response and stream chunk transforms to use strictOpenAiCompliance and transformFinishReason.
• src/providers/anthropic/index.ts: Added messagesCountTokens to Anthropic provider configuration.
• src/providers/azure-ai-inference/api.ts: Added imageEdit support and enhanced Entra ID authentication with azureEntraScope.
• src/providers/azure-ai-inference/index.ts: Added imageEdit configuration.
• src/providers/azure-ai-inference/utils.ts: Removed unused import.
• src/providers/azure-openai/api.ts: Added imageEdit support.
• src/providers/azure-openai/index.ts: Added imageEdit configuration.
• src/providers/bedrock/api.ts: Added messagesCountTokens endpoint mapping.
• src/providers/bedrock/chatComplete.ts: Updated content handling for messages to support video and S3 locations, added performance_config.
• src/providers/bedrock/complete.ts: Integrated transformFinishReason and TITAN_STOP_REASON for completion reasons.
• src/providers/bedrock/countTokens.ts: Added new configuration and response transform for Bedrock token counting.
• src/providers/bedrock/index.ts: Integrated messagesCountTokens and refined model string replacement.
• src/providers/bedrock/messages.ts: Added performance_config parameter.
• src/providers/bedrock/types.ts: Renamed BEDROCK_STOP_REASON to BEDROCK_CONVERSE_STOP_REASON, added TITAN_STOP_REASON, and updated content item types.
• src/providers/bedrock/uploadFile.ts: Enhanced model parameter handling to resolve foundation models from ARNs.
• src/providers/bedrock/utils/messagesUtils.ts: Refactored transformToolsConfig for cache control.
• src/providers/dashscope/api.ts: Updated base URL to international endpoint.
• src/providers/dashscope/index.ts: Added new parameters to chatComplete configuration.
• src/providers/deepseek/chatComplete.ts: Integrated transformFinishReason and DEEPSEEK_STOP_REASON for response and stream transforms.
• src/providers/deepseek/types.ts: Defined DEEPSEEK_STOP_REASON enum.
• src/providers/google-vertex-ai/api.ts: Added Mistral AI provider support and messagesCountTokens endpoint mapping.
• src/providers/google-vertex-ai/chatComplete.ts: Integrated transformFinishReason, modalities, and inlineData handling for image content.
• src/providers/google-vertex-ai/index.ts: Integrated messagesCountTokens and Mistral AI provider configuration.
• src/providers/google-vertex-ai/messagesCountTokens.ts: Added new configuration for Vertex AI token counting.
• src/providers/google-vertex-ai/transformGenerationConfig.ts: Added responseModalities transformation.
• src/providers/google-vertex-ai/types.ts: Added inlineData to ContentPart and defined VERTEX_GEMINI_GENERATE_CONTENT_FINISH_REASON enum.
• src/providers/google-vertex-ai/utils.ts: Updated provider checks and transformVertexLogprobs for new candidate types.
• src/providers/google/chatComplete.ts: Integrated transformFinishReason, GOOGLE_GENERATE_CONTENT_FINISH_REASON, modalities, and inlineData handling.
• src/providers/google/types.ts: Defined GOOGLE_GENERATE_CONTENT_FINISH_REASON enum.
• src/providers/index.ts: Added new provider configurations for Meshy, Tripo3D, and NextBit.
• src/providers/meshy/api.ts: Added new API configuration for Meshy.
• src/providers/meshy/index.ts: Added new provider configuration for Meshy.
• src/providers/mistral-ai/chatComplete.ts: Refactored response and stream transforms into factory functions, integrated transformFinishReason and MISTRAL_AI_FINISH_REASON.
• src/providers/mistral-ai/index.ts: Updated provider configuration to use new factory functions.
• src/providers/mistral-ai/types.ts: Defined MISTRAL_AI_FINISH_REASON enum.
• src/providers/nextbit/api.ts: Added new API configuration for NextBit.
• src/providers/nextbit/index.ts: Added new provider configuration for NextBit.
• src/providers/openai/api.ts: Added imageEdit support.
• src/providers/openai/chatComplete.ts: Standardized created timestamp format.
• src/providers/openai/complete.ts: Added stream_options parameter.
• src/providers/openai/index.ts: Added imageEdit configuration.
• src/providers/together-ai/chatComplete.ts: Integrated transformFinishReason and TOGETHER_AI_FINISH_REASON for response and stream transforms.
• src/providers/together-ai/types.ts: Defined TOGETHER_AI_FINISH_REASON enum.
• src/providers/tripo3d/api.ts: Added new API configuration for Tripo3D.
• src/providers/tripo3d/index.ts: Added new provider configuration for Tripo3D.
• src/providers/types.ts: Updated Endpoint and PROVIDER_FINISH_REASON types with new endpoints and stop reasons.
• src/providers/utils/finishReasonMap.ts: Expanded finish reason mappings for new provider-specific stop reasons.
• src/services/conditionalRouter.ts: Added url property to RouterContext.
• src/services/transformToProviderRequest.ts: Made providerOptions optional in transformUsingProviderConfig.
• src/types/requestBody.ts: Changed mistralFimCompletion type to string and added azureEntraScope.

🧪 Test Added

Extensive unit tests have been added for the new allowedRequestTypes plugin in plugins/default/default.test.ts. These tests cover various scenarios including allowing/blocking request types via parameters and metadata, handling comma-separated strings, streaming requests, and conflict detection between allow and block lists. The tests ensure correct verdict determination, error handling, and data output for the plugin. Other new handlers and provider integrations do not show explicit new tests in the provided JSON.

🔒Security Vulnerabilities

N/A

Motivation

To introduce new features, expand provider compatibility, enhance system robustness, and improve the overall flexibility and control of the AI gateway.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)

How Has This Been Tested?

• Unit Tests
• Integration Tests
• Manual Testing

Screenshots (if applicable)

N/A

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Related Issues

N/A

totalScore: 5

Tip

Quality Recommendations

1. Consider adding explicit unit or integration tests for the new imageEditsHandler and messagesCountTokensHandler to ensure their functionality and provider routing work as expected, similar to the comprehensive tests for allowedRequestTypes.

2. For the regexReplace plugin, if parameters.rule can be user-supplied, implement robust validation to prevent potential Regular Expression Denial of Service (ReDoS) attacks. This could involve limiting regex complexity or disallowing certain patterns.

3. Review the Dockerfile change removing apk update. While it might be for efficiency, ensure that apk upgrade --no-cache alone is sufficient to maintain necessary package versions and security patches without apk update refreshing the package lists, which could lead to outdated dependencies in some scenarios.

4. Ensure consistent error handling and logging across all new handlers and provider integrations. While some handlers have basic try-catch, a more standardized approach for error responses and detailed logging could improve observability and debugging.

Tanka Poem ♫

New plugins arise,
Gateway expands, strong and wide,
Code flows, swift and clean,
AI's power, now unbound,
Future's promise, brightly gleams. ✨

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway as Portkey AI Gateway
    participant ConditionalRouter
    participant HandlerUtils
    participant ProviderAPI as AI Provider API
    participant Plugin as New Plugins

    Client->>Gateway: POST /v1/images/edits (FormData)
    activate Gateway
    Gateway->>HandlerUtils: constructConfigFromRequestHeaders(headers)
    HandlerUtils-->>Gateway: camelCaseConfig

    Gateway->>Plugin: allowedRequestTypesHandler(context, params, 'beforeRequestHook')
    Plugin-->>Gateway: {verdict: true, data}

    Gateway->>HandlerUtils: tryTargetsRecursively(c, camelCaseConfig, request, headers, 'imageEdit', 'POST', 'config')
    activate HandlerUtils
    HandlerUtils->>ConditionalRouter: resolveTarget(currentTarget, {metadata, params, url})
    ConditionalRouter-->>HandlerUtils: finalTarget
    HandlerUtils->>ProviderAPI: POST /images/edits (transformed request)
    activate ProviderAPI
    ProviderAPI-->>HandlerUtils: 200 OK (Image Edit Response)
    deactivate ProviderAPI
    HandlerUtils-->>Gateway: tryTargetsResponse
    deactivate HandlerUtils

    Gateway->>Plugin: regexReplaceHandler(context, params, 'afterRequestHook')
    Plugin-->>Gateway: {verdict: true, transformedData}

    Gateway-->>Client: 200 OK (Image Edit Result)
    deactivate Gateway

    Client->>Gateway: POST /v1/messages/count_tokens (JSON)
    activate Gateway
    Gateway->>HandlerUtils: constructConfigFromRequestHeaders(headers)
    HandlerUtils-->>Gateway: camelCaseConfig

    Gateway->>Plugin: allowedRequestTypesHandler(context, params, 'beforeRequestHook')
    Plugin-->>Gateway: {verdict: true, data}

    Gateway->>HandlerUtils: tryTargetsRecursively(c, camelCaseConfig, request, headers, 'messagesCountTokens', 'POST', 'config')
    activate HandlerUtils
    HandlerUtils->>ConditionalRouter: resolveTarget(currentTarget, {metadata, params, url})
    ConditionalRouter-->>HandlerUtils: finalTarget
    HandlerUtils->>ProviderAPI: POST /model/{model}/count-tokens (transformed request)
    activate ProviderAPI
    ProviderAPI-->>HandlerUtils: 200 OK (Token Count Response)
    deactivate ProviderAPI
    HandlerUtils-->>Gateway: tryTargetsResponse
    deactivate HandlerUtils

    Gateway->>Plugin: modelRulesHandler(context, params, 'afterRequestHook')
    Plugin-->>Gateway: {verdict: true, data}

    Gateway-->>Client: 200 OK (Token Count Result)
    deactivate Gateway

Loading

[matter-code-review]

Added max_completion_tokens parameter mapping for Fireworks AI provider

[matter-code-review]

The pull request introduces several new features and improvements, including support for new providers, enhanced error handling, and additional configuration options. The changes appear well-structured and address the intended functionality.