Add support for dynamic usage of Flag Sets to `flagd`

⚠️ REJECTED IN FAVOR OF https://github.com/open-feature/flagd/blob/main/docs/architecture-decisions/duplicate-flag-keys.md ⚠️

The goal of this decision document is to establish flag sets as a first class concept in flagd, and support the dynamic addition/update/removal of flag sets at runtime.

Background

flagd is a language-agnostic feature flagging engine that forms a core part of the OpenFeature ecosystem. Flag configurations can be stored in different locations so called sources. These are specified at startup, e.g.:

flagd start \
  --port 8013 \
  --uri file:etc/flagd/my-flags-1.json \
  --uri https://my-flags-2.com/flags

The primary object here is to remove the coupling between sources and "logical" groups of flags, so that provider's aren't required to know their set of flags are sources from a file/http resource, etc, but could instead just supply a logical identifier for their flag set.

Requirements

Should enable the dynamic usage of flag sets as logical identifiers.
Should support configurations without flag sets.
Should adhere to existing OpenFeature and flagd terminology and concepts

Considered Options

Addition of flag set support in the flags schema and associated enhancements to flagd storage layer
Support for dynamically adding/removing flag sources through some kind of runtime configuration API
Support for dynamically adding/removing flag sources through some kind of "discovery" protocol or endpoint (ie: point flagd at a resource that would enumerate a mutable collection of secondary resources which represent flag sets)

Proposal

To support the dynamic usage of flag sets we propose to adapt the flag schema & storage layer in flagd. The changes will decouple flag sets from flag sources by supporting multiple flag sets within single flag sources. Dynamic updates to flag sources is already a feature of flagd.

New Schema Structure

The proposed changes to the current flagd schema would allow the following json structure for sources:

{
  "$schema": "https://flagd.dev/schema/v1/flagsets.json",
  "flagSets": {
    "my-project-1": {
      "metadata": {
        ...
      },
      "flags": {
        "my-flag-1": {
          "metadata": {
            ...
          },
          ...
        },
        ...
      },
      "$evaluators": {
        ...
      }
    },
    "my-project-2": {
      ...
    }
  }
}

We propose to introduce a 3rd json schema flagSets.json, which references to flags.json:

flagSets.json (new)
flags.json
targeting.json

We don't want to support merging of flag sets, due to implementation efforts & potential confusing behaviour of the merge strategy. Therefore, we propose for the initial implementation, flagSetIds must be unique across different sources or the configuration is considered invalid. In the future, it might be useful to support and implement multiple "strategies" for merging flagSets from different sources, but that's beyond the scope of this proposal.

New Data Structure

The storage layer in flagd requires refactoring to better support multiple flag sets within one source.

package store

type State struct {
    FlagSets map[string]FlagSet `json:"flagSets"` // key = flagSetId
}

type FlagSet struct {
    Flags    map[string]model.Flag `json:"flags"` // key = flagKey
    Metadata Metadata              `json:"metadata,omitempty"`
}

type Flag struct {
    State          string          `json:"state"`
    DefaultVariant string          `json:"defaultVariant"`
    Variants       map[string]any  `json:"variants"`
    Targeting      json.RawMessage `json:"targeting,omitempty"`
    Metadata       Metadata        `json:"metadata,omitempty"`
}

type Metadata = map[string]interface{}

OpenFeature Provider Implications

Currently, creating a new flagd provider can look like follows:

final FlagdProvider flagdProvider =
        new FlagdProvider(FlagdOptions.builder()
                .resolverType(Config.Evaluator.IN_PROCESS)
                .host("localhost")
                .port(8015)
                .selector("myFlags.json")
                .build());

With the proposed solution the flagSetId should be passed to the builder as selector argument instead of the source.
null is now a valid selector value, referencing flags which do not belong to a flag set. The default/fallback flagSetId should be null.

Consequences

Good, because it decouples flag sets from the sources
Good, because we will refactor the flagd storage layer (which is currently storing duplicate data & difficult to understand)
Good, because we can support backwards compatibility with the v0 schema
Good, because the "null" flag set is logically treated as any other flag set, reducing overall implementation complexity.
Bad, because there's additional complexity to support this new config schema as well as the current.
Bad, because this is a breaking change in the behavior of the selector member.

Other Options

We evaluated the mentioned options as follows: options 2 + 3: support for dynamically adding/removing flag sources and decided against this option because it requires much more implementation effort than option 1. Required changes include:

flagd/core/sync: dynamic mode, which allows specifying the sync type that should be added/removed at runtime
flagd/flagd: startup dynamic sync configuration
make sure to still support static syncs

More Information

Current flagd schema: flags.json
flagd storage layer implementation: store/flags.go
flagd GitHub Repository
OpenFeature Project Overview

6.0 KiB Raw Permalink Blame History

Add support for dynamic usage of Flag Sets to flagd