Configure observability

An HTTP request and a completed business action are different observations. A request monitor can show that POST /notes returned errors. A counter can record that a note was actually saved. Choose the signal that answers your operational question.

#Understand: observe, count, and optionally forward

Premium observability currently targets GCP Cloud Run. Customer services collect configured HTTP observations and application counters. This is not a general tracing system or a store for arbitrary raw logs.

Declaring a counter does not increment it. A failed metric call after a successful database commit does not undo that commit. Decide how to report telemetry failure without repeating the business operation.

#Configure: Pub/Sub failures and dead letters

Declare native delivery checks in the same application observability YAML:

apiVersion: fluffy-chainsaw.dev/observability/v1
runtimes:
  api:
    pubsubMonitors:
      email-delivery:
        name: Email delivery
        environments:
          prod:
            criticality: critical
            hostedForwarding: true
            subscription: projects/example-project/subscriptions/email-push
            deadLetterSubscription: projects/example-project/subscriptions/email-dlq-retained
            # Optional: create the retention subscription on this existing topic.
            deadLetterTopic: projects/example-project/topics/email-dlq

subscription and deadLetterSubscription are required, distinct full GCP resource names in the installation project. The source subscription must already have a dead-letter policy. Include deadLetterTopic only when Premium should create the named retention subscription; omit it when that subscription is already managed elsewhere. Creating a retention subscription does not configure the source's dead-letter policy.

hostedForwarding: true inside this monitor's environment sends native incidents to Fluffy. It does not enable the periodic HTTP/counter collector. If it is omitted, application-wide forwarding consent also applies; with no consent, incidents remain in the customer's GCP project.

Each monitor creates five native checks:

The generated policies use push_request_count, dead_letter_message_count, num_undelivered_messages, and oldest_unacked_message_age. Forwarding errors use the response_code label on dead_letter_message_count; the GCP metric catalog does not provide a separate dead_letter_publish_error_count metric. See Google's Pub/Sub monitoring documentation.

Cloud Monitoring evaluates these conditions and pushes incident notifications. There is no application timer polling the metrics. Fluffy displays failures as Error and closed incidents as Recovered, with a link to Cloud Monitoring. Notifications and metrics can arrive after the underlying event. Duplicate and out-of-order notifications do not reopen a recovered incident.

A retained subscription keeps failed messages for seven days without an idle consumer. Inspect and repair the cause before replaying. Preserve the original message and business ID when republishing, then acknowledge the DLQ delivery only after publication succeeds. Republish the original message.data, attributes and ordering key, preserving the business ID; do not publish the entire pull-response envelope as application data. GCP adds CloudPubSubDeadLetterSource* attributes identifying the failed delivery. See dead-letter topics and messaging delivery guarantees.

The incident notification pipeline has its own retained DLQ and an independent Cloud Monitoring backlog policy. If that pipeline is unavailable, inspect Cloud Monitoring directly; a broken delivery channel cannot notify through itself.

#Configure: observe note creation

Start with an application deployed to a declared prod environment and Premium installed with observability enabled. The following example has one runtime, api, which serves POST /notes.

Beside its application manifest, create fluffy-chainsaw.observability.yaml:

apiVersion: fluffy-chainsaw.dev/observability/v1
runtimes:
  api:
    httpMonitors:
      create-note:
        name: Create note requests
        path: /notes
        methods: [POST]
        environments:
          prod: {criticality: standard}
    counters:
      notes-created:
        name: Notes created

The runtime keys must exactly match the adjacent application. If it also has a worker with no monitors or counters, include worker: {}. Environment references must name declared application environments.

#Grant the counter operation

Register the reviewed extension executable, then merge these declarations into the application:

extensions:
  premium:
    environments: [local, prod]
    configuration: [fluffy-chainsaw.observability.yaml]
runtimes:
  api:
    uses:
      extensions:
        premium:
          - capability: counters
            name: notes-created
            actions: [increment]

The extension configuration defines the counter; the runtime grant selects who can change it. Generate the API's package:

fluffy-chainsaw generate --application example-app --runtime api --language typescript

Use the sidecar setup for address and transport (TypeScript), channel (Python), or the Go connection. Imports below assume application example-app:

import { Metadata } from '@grpc/grpc-js'
import * as runtimev1 from './.fluffy/generated/runtimes/api/typescript/fluffy/runtime/example-app/api.js'
import * as countersv1 from './.fluffy/generated/runtimes/api/typescript/extensions/premium/counters/v1/counters.js'

const client = new runtimev1.PremiumCountersNotesCreatedClient(address, transport)
const result = await new Promise<unknown>((resolve, reject) => {
  client.increment(
    countersv1.ChangeRequest.fromPartial({ amount: 1 }),
    new Metadata(),
    { deadline: Date.now() + 5000 },
    (error, value) => {
      if (error) reject(error)
      else resolve(value)
    },
  )
})

Call this after the note commits, using your chosen telemetry-error policy. There is no separate HTTP counter endpoint or environment variable to invent.

#Add an alert

For example, add an alert under the existing counter to detect no recorded increases over fifteen minutes:

notes-created:
  name: Notes created
  alerts:
    no-new-notes:
      environments: [prod]
      criticality: best-effort
      count:
        direction: increase
        operator: eq
        threshold: 0
        window: 15m

This is appropriate only if normal traffic makes that condition meaningful. A quiet application and a broken application can both produce no observations; a threshold alone cannot distinguish them. Keep the alert under runtimes.api.counters, rather than adding a second root mapping.

#Choose whether to forward hosted observations

Customer collection and hosted forwarding are separate choices. To enable forwarding for this application's production environment, merge:

environments:
  prod:
    hostedForwarding: true

Omitting or disabling this setting opts out on the next deployment. Customer-side collection can continue. The defined minimized monitoring data crosses the hosted boundary; raw logs remain in the customer project. Optional Slack configuration belongs to installation integrations.

#Deploy and verify

Preserve existing installation capabilities while enabling capabilities.observability: true. Preview and apply the Premium installation changes, then use the Core application deployment workflow.

Create a note successfully and inspect the matching customer counter and HTTP monitor. Also exercise a controlled invalid request and distinguish its HTTP result from successful creation. Hosted views require forwarding, entitlement, and enough settled traffic; missing data is not a healthy result.

Local extension execution disables cloud forwarding. Use it to check SDK wiring, not to expect production Cloud Run metrics.

#Reference: observability fields and limits

The document uses apiVersion: fluffy-chainsaw.dev/observability/v1, runtimes, and optional environments.<name>.hostedForwarding. Unknown fields are rejected. Every adjacent application runtime needs an entry, even if empty.

Configuration supports at most 100 resolved HTTP monitors and 1,000 counters, with at most 100 alerts per counter. An alert selects 1–100 declared environments. Monitor IDs must be unique across resolved configuration, including environments; use distinct IDs when configuring independent environment monitors. Counter IDs are scoped to application/runtime. Names are nonempty, trimmed, and at most 100 characters.

Counter grants support increment and decrement; only grant the actions the runtime uses. Generated SDKs explains target-specific methods.

#Diagnose missing observations

Rebuilding only the companion executable does not replace an older extension runtime image. Local service archives are not automatically consumed by the extension image loader. Check the installed image when diagnosing a protocol mismatch.

Continue to uninstall and export for changing ownership or removing customer services.