Ecosystem
All entries on this page were added by people who worked on these and thus self-identified as being part of the Project Capsule Ecosystem.
Integrations
Capsule works well with other CNCF kubernetes based solutions. Below you can see the ones we have documented. In the end it can work with any solution, due to Capsule’s kubernetes native approach:
Addons
Addons are separate projects which interact with the core Capsule Project. Since our commitment is, to have a stable core API we decided to push towards an addon based ecosystem. If you have a new addon, which interacts with the capsule core project, consider adding the addon.
Proxy
core
ux
Enhance the user experience by allowing users to query the Kubernetes API and only getting the results, they are supposed to get.
Rancher
community
ux
Integrate Capsule with Rancher to manage Capsule Tenants and their resources with Rancher Projects.
ArgoCD
vendor
gitops
This addon is designed for kubernetes administrators, to automatically translate their existing Capsule Tenants into Argo Appprojects.
Sops Operator
core
secrets
gitops
Handle SOPS Secrets in a multi-tenant and kubernetes-native way.
FluxCD
core
gitops
In particular enables Tenants to manage their resources, including creating Namespaces.
Cortex Proxy
core
observability
Route metrics to cortex organizations based on the relational of namespace metrics to capsule tenants.
1 - Integrations
Integrate Capsule with other platforms and solutions
1.1 - ArgoCD
Capsule Integration with ArgoCD
Integration
Resource Actions
You may provide Custom Resource Actions for Capsule specific resources and interactions.
Namespace Resource Actions
With the following configuration, ArgoCD will show Cordon and Resume actions for the Namespace resource. The Cordon action will set the projectcapsule.dev/cordoned label to true, while the Resume action will set it to false. This is only for Namespaces part of a Capsule Tenant.
resource.customizations.actions.Namespace: |
mergeBuiltinActions: true
discovery.lua: |
actions = {
cordon = {
iconClass = "fa fa-solid fa-pause",
disabled = true,
},
uncordon = {
iconClass = "fa fa-solid fa-play",
disabled = true,
},
}
local function has_managed_ownerref()
if obj.metadata == nil or obj.metadata.ownerReferences == nil then
return false
end
for _, ref in ipairs(obj.metadata.ownerReferences) do
if ref.kind == "Tenant" and ref.apiVersion == "capsule.clastix.io/v1beta2" then
return true
end
end
return false
end
if not has_managed_ownerref() then
return {}
end
local labels = {}
if obj.metadata ~= nil and obj.metadata.labels ~= nil then
labels = obj.metadata.labels
end
local cordoned = labels["projectcapsule.dev/cordoned"] == "true"
if cordoned then
actions["uncordon"].disabled = false
else
actions["cordon"].disabled = false
end
return actions
definitions:
- name: cordon
action.lua: |
if obj.metadata == nil then
obj.metadata = {}
end
if obj.metadata.labels == nil then
obj.metadata.labels = {}
end
obj.metadata.labels["projectcapsule.dev/cordoned"] = "true"
return obj
- name: uncordon
action.lua: |
if obj.metadata ~= nil and obj.metadata.labels ~= nil then
obj.metadata.labels["projectcapsule.dev/cordoned"] = "false"
end
return obj
Tenant Resource Actions
With the following configuration, ArgoCD will show Cordon and Resume actions for the Tenant resource. The Cordon action will set the spec.cordon field to true, while the Resume action will set it to false.
resource.customizations.actions.capsule.clastix.io_Tenant: |
mergeBuiltinActions: true
discovery.lua: |
actions = {}
actions["cordon"] = {
["iconClass"] = "fa fa-solid fa-pause",
["disabled"] = true,
}
actions["uncordon"] = {
["iconClass"] = "fa fa-solid fa-play",
["disabled"] = true,
}
local suspend = false
if obj.spec ~= nil and obj.spec.cordoned ~= nil then
suspend = obj.spec.cordoned
end
if suspend then
actions["uncordon"]["disabled"] = false
else
actions["cordon"]["disabled"] = false
end
return actions
definitions:
- name: cordon
action.lua: |
if obj.spec == nil then
obj.spec = {}
end
obj.spec.cordoned = true
return obj
- name: uncordon
action.lua: |
if obj.spec ~= nil and obj.spec.cordoned ~= nil and obj.spec.cordoned then
obj.spec.cordoned = false
end
return obj
TenantResource and GlobalTenantResource Actions
With the following configuration, ArgoCD will show Cordon, Uncordon, and Reconcile actions for TenantResource and GlobalTenantResource. Cordoning pauses all apply and delete operations. The Reconcile action triggers an immediate reconcile by setting the reconcile.projectcapsule.dev/requestedAt annotation. Both Cordon and Reconcile are disabled when the resource is already cordoned, since the controller skips processing in that state.
resource.customizations.actions.capsule.clastix.io_TenantResource: |
mergeBuiltinActions: true
discovery.lua: |
local actions = {}
actions["cordon"] = {
["iconClass"] = "fa fa-fw fa-pause",
["disabled"] = true,
}
actions["uncordon"] = {
["iconClass"] = "fa fa-fw fa-play",
["disabled"] = true,
}
actions["reconcile"] = {
["iconClass"] = "fa fa-fw fa-rotate-right",
["disabled"] = true,
}
local cordoned = false
if obj.spec ~= nil and obj.spec.cordoned ~= nil then
cordoned = obj.spec.cordoned
end
if cordoned then
actions["uncordon"]["disabled"] = false
else
actions["cordon"]["disabled"] = false
actions["reconcile"]["disabled"] = false
end
return actions
definitions:
- name: cordon
action.lua: |
if obj.spec == nil then
obj.spec = {}
end
obj.spec.cordoned = true
return obj
- name: uncordon
action.lua: |
if obj.spec ~= nil and obj.spec.cordoned ~= nil and obj.spec.cordoned then
obj.spec.cordoned = false
end
return obj
- name: reconcile
action.lua: |
local os = require("os")
if obj.metadata.annotations == nil then
obj.metadata.annotations = {}
end
obj.metadata.annotations["reconcile.projectcapsule.dev/requestedAt"] = os.date("!%Y-%m-%dT%XZ")
return obj
Apply the same block for GlobalTenantResource by replacing the resource key:
resource.customizations.actions.capsule.clastix.io_GlobalTenantResource: |
# same content as above
Resource Health
You may provide Custom Resource Health for Capsule specific resources and interactions.
Tenant Resource Health
Shows Suspended when the Tenant is cordoned, reflecting that no new workloads can be scheduled in its Namespaces. Reports Degraded when the Ready condition is False, Healthy when Ready is True, and Progressing otherwise.
resource.customizations.health.capsule.clastix.io_Tenant: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Cordoned" and condition.status == "True" then
hs.status = "Suspended"
hs.message = condition.message
return hs
end
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
Namespace Resource Health
Suspends a Namespace when it’s Cordoned. This is only for Namespaces part of a Capsule Tenant.
resource.customizations.health.Namespace: |
local hs = {}
local function has_managed_ownerref()
if obj.metadata == nil or obj.metadata.ownerReferences == nil then
return false
end
for _, ref in ipairs(obj.metadata.ownerReferences) do
if ref.kind == "Tenant" and ref.apiVersion == "capsule.clastix.io/v1beta2" then
return true
end
end
return false
end
local labels = {}
if obj.metadata ~= nil and obj.metadata.labels ~= nil then
labels = obj.metadata.labels
end
local cordoned = labels["projectcapsule.dev/cordoned"] == "true"
if cordoned and has_managed_ownerref() then
hs.status = "Suspended"
hs.message = "Namespace is cordoned (tenant-managed)"
return hs
end
if obj.status ~= nil and obj.status.phase ~= nil then
if obj.status.phase == "Active" then
hs.status = "Healthy"
hs.message = "Namespace is Active"
return hs
else
hs.status = "Progressing"
hs.message = "Namespace phase is " .. obj.status.phase
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Namespace status"
return hs
CapsuleConfiguration Resource Health
Reports health based on the Ready condition.
resource.customizations.health.capsule.clastix.io_CapsuleConfiguration: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
TenantOwner Resource Health
Reports Degraded when the TenantOwner failed to reconcile, and Healthy when the owner has been successfully bound to its tenant.
resource.customizations.health.capsule.clastix.io_TenantOwner: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
ResourcePool Resource Health
Reports Degraded when any resource is exhausted or not ready, and Healthy when the pool is active and within limits.
resource.customizations.health.capsule.clastix.io_ResourcePool: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
if obj.status.exhaustions ~= nil then
local exhausted = {}
for resource, _ in pairs(obj.status.exhaustions) do
table.insert(exhausted, resource)
end
table.sort(exhausted)
if #exhausted > 0 then
hs.status = "Degraded"
hs.message = "Pool exhausted for: " .. table.concat(exhausted, ", ")
return hs
end
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
ResourcePoolClaim Resource Health
Reports Suspended when unbound (waiting for a pool), Degraded when not ready, and Healthy when bound and ready.
resource.customizations.health.capsule.clastix.io_ResourcePoolClaim: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Bound" and condition.status == "False" then
hs.status = "Suspended"
hs.message = condition.message
return hs
end
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
CustomQuota Resource Health
Reports Degraded when the quota reconciliation failed (e.g. a matched resource has a missing field), and Healthy when usage has been successfully calculated for the namespace.
resource.customizations.health.capsule.clastix.io_CustomQuota: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
GlobalCustomQuota Resource Health
Reports Degraded when the quota reconciliation failed, and Healthy when usage has been successfully calculated across all selected namespaces.
resource.customizations.health.capsule.clastix.io_GlobalCustomQuota: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
GlobalResourceQuota Resource Health
Reports Progressing when initialization or reconciliation is in progress (e.g. Ready condition reason Reconciling), Degraded when the resource quota reconciliation failed, and Healthy when usage has been successfully calculated across all selected namespaces.
resource.customizations.health.capsule.clastix.io_GlobalResourceQuota: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" then
if condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
elseif condition.status == "False" then
if condition.reason == "Reconciling" then
hs.status = "Progressing"
hs.message = condition.message
return hs
end
hs.status = "Degraded"
hs.message = condition.message
return hs
else
hs.status = "Progressing"
hs.message = condition.message
return hs
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
TenantResource Resource Health
Reports Suspended when the replication is cordoned (paused for maintenance). Reports Degraded when the replication of tenant-scoped resources failed, and Healthy when all resources have been successfully replicated into the target namespaces.
resource.customizations.health.capsule.clastix.io_TenantResource: |
local hs = {}
if obj.spec ~= nil and obj.spec.cordoned ~= nil and obj.spec.cordoned then
hs.status = "Suspended"
hs.message = "Replication is cordoned"
return hs
end
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
GlobalTenantResource Resource Health
Reports Suspended when the replication is cordoned (paused for maintenance). Reports Degraded when the cluster-wide resource replication failed, and Healthy when all resources have been successfully replicated across all tenant namespaces.
resource.customizations.health.capsule.clastix.io_GlobalTenantResource: |
local hs = {}
if obj.spec ~= nil and obj.spec.cordoned ~= nil and obj.spec.cordoned then
hs.status = "Suspended"
hs.message = "Replication is cordoned"
return hs
end
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
Capsule Proxy
The following health checks apply to Capsule Proxy CRDs.
ProxySetting Resource Health
Reports Degraded when a per-user or per-group ProxySetting failed to reconcile, and Healthy when the proxy rules have been successfully applied.
resource.customizations.health.capsule.clastix.io_ProxySetting: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
GlobalProxySettings Resource Health
Reports Degraded when the cluster-wide proxy settings failed to reconcile, and Healthy when the global proxy rules have been successfully applied.
resource.customizations.health.capsule.clastix.io_GlobalProxySettings: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" and condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.type == "Ready" and condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
SOPS Operator
The following health checks apply to SOPS Operator CRDs (addons.projectcapsule.dev), which is a Capsule Addon.
SopsSecret Resource Health
Reports Degraded when decryption or secret replication failed, and Healthy when all managed secrets have been successfully decrypted and replicated.
resource.customizations.health.addons.projectcapsule.dev_SopsSecret: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" then
if condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
if condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.status == "Unknown" then
hs.status = "Progressing"
hs.message = condition.message
return hs
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
SopsProvider Resource Health
Reports Degraded when one or more decryption providers failed to load or validate, and Healthy when all configured providers are available.
resource.customizations.health.addons.projectcapsule.dev_SopsProvider: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" then
if condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
if condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.status == "Unknown" then
hs.status = "Progressing"
hs.message = condition.message
return hs
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
GlobalSopsSecret Resource Health
Reports Degraded when decryption or cross-namespace secret replication failed, and Healthy when all managed secrets have been successfully decrypted and replicated into their target namespaces.
resource.customizations.health.addons.projectcapsule.dev_GlobalSopsSecret: |
local hs = {}
if obj.status == nil or obj.status.conditions == nil then
hs.status = "Progressing"
hs.message = "Waiting for status"
return hs
end
if obj.metadata ~= nil and obj.metadata.generation ~= nil and obj.status.observedGeneration ~= nil
and obj.status.observedGeneration ~= obj.metadata.generation then
hs.status = "Progressing"
hs.message = "Waiting for reconciliation (generation mismatch)"
return hs
end
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" then
if condition.status == "True" then
hs.status = "Healthy"
hs.message = condition.message
return hs
end
if condition.status == "False" then
hs.status = "Degraded"
hs.message = condition.message
return hs
end
if condition.status == "Unknown" then
hs.status = "Progressing"
hs.message = condition.message
return hs
end
end
end
hs.status = "Progressing"
hs.message = "Waiting for Ready condition"
return hs
1.2 - Crossplane
Capsule Integration with Crossplane
1.3 - Dashboard
Capsule Integration with Kubernetes Dashboard
This guide works with the kubernetes dashboard v2.0.0 (Chart 6.0.8). It has not yet been tested successfully with with v3.x version of the dashboard.
We recommend to use Headlamp as a more modern alternative to the Kubernetes Dashboard.
This guide describes how to integrate the Kubernetes Dashboard and Capsule Proxy with OIDC authorization.
OIDC Authentication
Your cluster must also be configured to use OIDC Authentication for seamless Kubernetes RBAC integration. In a such scenario, you should have in the kube-apiserver.yaml manifest the following content:
spec:
containers:
- command:
- kube-apiserver
...
- --oidc-issuer-url=https://${OIDC_ISSUER}
- --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
- --oidc-client-id=${OIDC_CLIENT_ID}
- --oidc-username-claim=preferred_username
- --oidc-groups-claim=groups
- --oidc-username-prefix=-
Where ${OIDC_CLIENT_ID} refers to the client ID that all tokens must be issued.
For this client we need: 1. Check Valid Redirect URIs: in the oauth2-proxy configuration we set redirect-url: “https://${DASHBOARD_URL}/oauth2/callback”, it needs to add this path to the Valid Redirect URIs 2. Create a mapper with Mapper Type ‘Group Membership’ and Token Claim Name ‘groups’. 3. Create a mapper with Mapper Type ‘Audience’ and Included Client Audience and Included Custom Audience set to your client name (${OIDC_CLIENT_ID}).
OAuth2 Proxy
To enable the proxy authorization from the Kubernetes dashboard to Keycloak, we need to use an OAuth proxy. In this article, we will use oauth2-proxy and install it as a pod in the Kubernetes Dashboard namespace. Alternatively, we can install oauth2-proxy in a different namespace or use it as a sidecar container in the Kubernetes Dashboard deployment.
Prepare the values for oauth2-proxy:
cat > values-oauth2-proxy.yaml <<EOF
config:
clientID: "${OIDC_CLIENT_ID}"
clientSecret: ${OIDC_CLIENT_SECRET}
extraArgs:
provider: "keycloak-oidc"
redirect-url: "https://${DASHBOARD_URL}/oauth2/callback"
oidc-issuer-url: "https://${KEYCLOAK_URL}/auth/realms/${OIDC_CLIENT_ID}"
pass-access-token: true
set-authorization-header: true
pass-user-headers: true
ingress:
enabled: true
path: "/oauth2"
hosts:
- ${DASHBOARD_URL}
tls:
- hosts:
- ${DASHBOARD_URL}
EOF
More information about the keycloak-oidc provider can be found on the oauth2-proxy documentation. We’re ready to install the oauth2-proxy:
helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install oauth2-proxy oauth2-proxy/oauth2-proxy -n ${KUBERNETES_DASHBOARD_NAMESPACE} -f values-oauth2-proxy.yaml
Configuring Keycloak
The Kubernetes cluster must be configured with a valid OIDC provider: for our guide, we’re giving for granted that Keycloak is used, if you need more info please follow the OIDC Authentication section.
In a such scenario, you should have in the kube-apiserver.yaml manifest the following content:
spec:
containers:
- command:
- kube-apiserver
...
- --oidc-issuer-url=https://${OIDC_ISSUER}
- --oidc-ca-file=/etc/kubernetes/oidc/ca.crt
- --oidc-client-id=${OIDC_CLIENT_ID}
- --oidc-username-claim=preferred_username
- --oidc-groups-claim=groups
- --oidc-username-prefix=-
Where ${OIDC_CLIENT_ID} refers to the client ID that all tokens must be issued.
For this client we need:
- Check
Valid Redirect URIs: in the oauth2-proxy configuration we set redirect-url: "https://${DASHBOARD_URL}/oauth2/callback", it needs to add this path to the Valid Redirect URIs - Create a mapper with Mapper Type ‘Group Membership’ and Token Claim Name ‘groups’.
- Create a mapper with Mapper Type ‘Audience’ and Included Client Audience and Included Custom Audience set to your client name(OIDC_CLIENT_ID).
Configuring Kubernetes Dashboard
If your Capsule Proxy uses HTTPS and the CA certificate is not the Kubernetes CA, you need to add a secret with the CA for the Capsule Proxy URL.
cat > ca.crt<< EOF
-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----
EOF
kubectl create secret generic certificate --from-file=ca.crt=ca.crt -n ${KUBERNETES_DASHBOARD_NAMESPACE}
Prepare the values for the Kubernetes Dashboard:
cat > values-kubernetes-dashboard.yaml <<EOF
extraVolumes:
- name: token-ca
projected:
sources:
- serviceAccountToken:
expirationSeconds: 86400
path: token
- secret:
name: certificate
items:
- key: ca.crt
path: ca.crt
extraVolumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: token-ca
ingress:
enabled: true
annotations:
nginx.ingress.kubernetes.io/auth-signin: https://${DASHBOARD_URL}/oauth2/start?rd=$escaped_request_uri
nginx.ingress.kubernetes.io/auth-url: https://${DASHBOARD_URL}/oauth2/auth
nginx.ingress.kubernetes.io/auth-response-headers: "authorization"
hosts:
- ${DASHBOARD_URL}
tls:
- hosts:
- ${DASHBOARD_URL}
extraEnv:
- name: KUBERNETES_SERVICE_HOST
value: '${CAPSULE_PROXY_URL}'
- name: KUBERNETES_SERVICE_PORT
value: '${CAPSULE_PROXY_PORT}'
EOF
To add the Certificate Authority for the Capsule Proxy URL, we use the volume token-ca to mount the ca.crt file. Additionally, we set the environment variables KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to route requests to the Capsule Proxy.
Now you can install the Kubernetes Dashboard:
helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard -n ${KUBERNETES_DASHBOARD_NAMESPACE} -f values-kubernetes-dashboard.yaml
1.4 - Envoy Gateway
Capsule Integration with Envoy (Gateway API)
With Envoy Gateway, tenants can publish their
applications through the Gateway API while the
platform manages the Gateway infrastructure. This guide covers protecting
platform Gateways, generating a managed Gateway per Tenant, and delegating
listeners with Capsule hostname rules. Before starting, review the upstream
documentation:
The examples assume Envoy Gateway is installed and a tenants GatewayClass is
available. Examples using allowedListeners also require Gateway API CRDs and
an Envoy Gateway release supporting gateway.networking.k8s.io/v1 ListenerSets.
Secure Gateways
For Gateways that are reserved for platform services, exclude Tenant namespaces
from allowedRoutes. Capsule labels Tenant namespaces with
capsule.clastix.io/tenant, so a DoesNotExist selector allows routes only from
namespaces outside Capsule Tenants. Disable delegated listeners with
allowedListeners.namespaces.from: None:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: platform-gateway
namespace: platform-system
spec:
gatewayClassName: tenants
allowedListeners:
namespaces:
from: None
listeners:
- name: http
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Selector
selector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: DoesNotExist
Keep these Gateways and their infrastructure in platform-owned namespaces that
Tenant owners cannot modify. For Gateways intended for a Tenant, select that
Tenant’s namespace label instead, as shown below. Capsule
ingress rules then restrict the
hostnames used by resources inside those namespaces.
Managed Gateways
One approach is to generate a dedicated Gateway for each Tenant. The platform
creates a tenant-<tenant-name>-system namespace outside the Tenant, where it
manages the Gateway, EnvoyProxy, and TLS certificates. Tenant owners manage
HTTPRoutes in their own namespaces:
Gateway generation
The following GlobalTenantResource generates the
managed Gateway and EnvoyProxy. The EnvoyProxy configures the Envoy instances;
the Gateway defines listeners and their allowed routes. Create the platform
namespaces and the tenant-critical PriorityClass before using this example.
The resource collects HTTPRoutes labeled
projectcapsule.dev/tenant: <tenant-name> and generates an HTTPS listener for
each hostname. Tenant data can supply
bandwidth limits and a load balancer address. The Gateway annotations reference
the namespaced Issuer configured under Certificate management.
---
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: managed-envoy-gateway
spec:
scope: Tenant
resyncPeriod: 30s
resources:
- context:
resources:
- apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
index: https
selector:
matchLabels:
projectcapsule.dev/tenant: "{{tenant.name}}"
generators:
- missingKey: zero
template: |
{{- $ingressBandwidth := dig "spec" "data" "networking" "ingress" "bandwidth" "" $.tenant }}
{{- $egressBandwidth := dig "spec" "data" "networking" "egress" "bandwidth" "" $.tenant }}
{{- $loadBalancerIP := dig "spec" "data" "networking" "ingress" "loadbalancer" "" $.tenant }}
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: tenant-{{ $.tenant.metadata.name }}-gateway
namespace: tenant-{{ $.tenant.metadata.name }}-system
spec:
logging:
level:
default: info
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 2
pod:
priorityClassName: tenant-critical
{{- if or $ingressBandwidth $egressBandwidth }}
annotations:
{{- with $ingressBandwidth }}
kubernetes.io/ingress-bandwidth: {{ . }}
{{- end }}
{{- with $egressBandwidth }}
kubernetes.io/egress-bandwidth: {{ . }}
{{- end }}
{{- end }}
{{- with $loadBalancerIP }}
envoyService:
loadBalancerIP: {{ . }}
{{- end }}
- missingKey: zero
template: |
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: tenant-{{ $.tenant.metadata.name }}-gateway
namespace: tenant-{{ $.tenant.metadata.name }}-system
annotations:
cert-manager.io/issuer: tenant-{{ $.tenant.metadata.name }}-acme-http
cert-manager.io/private-key-size: "4096"
cert-manager.io/private-key-algorithm: RSA
spec:
gatewayClassName: tenants
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: tenant-{{ $.tenant.metadata.name }}-gateway
listeners:
- name: http-challenge
port: 80
protocol: HTTP
allowedRoutes:
namespaces:
from: Same
{{- range $_, $http := $.https }}
{{- range $i, $hostname := $http.spec.hostnames }}
- name: {{ $http.metadata.namespace }}-{{ $http.metadata.name }}-{{ $i }}
port: 443
protocol: HTTPS
hostname: {{ $hostname }}
tls:
mode: Terminate
certificateRefs:
- group: ''
kind: Secret
name: {{ $http.metadata.namespace }}-{{ $http.metadata.name }}-{{ $i }}-tls
allowedRoutes:
namespaces:
from: Selector
selector:
matchLabels:
kubernetes.io/metadata.name: "{{ $http.metadata.namespace }}"
{{- end }}
{{- end }}
Certificate management
To manage certificates with cert-manager, enable
its Gateway API support and generate an Issuer in each platform namespace.
This Issuer uses ACME HTTP-01 challenges
through the managed Gateway’s http-challenge listener, which accepts solver
routes from the same platform namespace. The
Gateway annotations above let
cert-manager create the certificates for its generated HTTPS listeners:
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: tenant-acme-issuer
spec:
scope: Tenant
resources:
- rawItems:
- apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: "tenant-{{tenant.name}}-acme-http"
namespace: "tenant-{{tenant.name}}-system"
spec:
acme:
email: platform@email.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cert-letsencrypt-staging
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: "tenant-{{tenant.name}}-gateway"
namespace: "tenant-{{tenant.name}}-system"
sectionName: http-challenge
Tenant ListenerSets
Another approach is to let tenants define listeners and certificates while the
platform manages the Gateway and its Envoy infrastructure. The Gateway allows
ListenerSets from the Tenant’s namespaces;
a Capsule ingress rule restricts their hostnames to subdomains of
solar.example.com.
Reuse the managed EnvoyProxy and tenants GatewayClass from
Managed Gateways, and replace the generated Gateway
configuration with the one below. If using a GlobalTenantResource, update its
Gateway generator and remove the HTTPRoute context lookup and the loop that
generates HTTPS listeners. Tenants now provide those listeners themselves.
Tenant Gateway default and hostname rule
The platform administrator configures the solar Tenant with a default Gateway
reference for ListenerSets and an allow rule for both ListenerSet and HTTPRoute
hostnames:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: User
name: alice
rules:
- mutate:
- ingress:
listenerSets:
parentRef:
name: "tenant-{{ .tenant.metadata.name }}-gateway"
namespace: "tenant-{{ .tenant.metadata.name }}-system"
enforce:
action: allow
ingress:
types:
- ListenerSet
- HTTPRoute
hostnames:
- exp: '^.+\.solar\.example\.com$'
The rule applies to every namespace belonging to solar. The
ListenerSet mutation fills a missing spec.parentRef
with tenant-solar-gateway in tenant-solar-system. action defaults to merge;
an explicit parent reference is retained. group defaults to
gateway.networking.k8s.io and kind to Gateway.
Capsule checks each listener’s hostname and every entry in an HTTPRoute’s
spec.hostnames during
creation and updates. The anchored expression allows hostnames at any depth
below solar.example.com, including wildcard hostnames within that domain.
It excludes the parent hostname solar.example.com itself. Targeting both
resource types ensures that the same policy also applies to routes attached
directly to the Gateway.
| Hostname | Rule result |
|---|
web.solar.example.com | Allowed |
api.prod.solar.example.com | Allowed |
*.solar.example.com | Allowed |
solar.example.com | Denied |
web.lunar.example.com | Denied |
web.solar.example.com.evil.com | Denied |
Allow tenant ListenerSets on the managed Gateway
The platform owns this Gateway in tenant-solar-system, outside the Tenant.
allowedListeners selects namespaces by Capsule’s tenant label, allowing
ListenerSets from solar namespaces to attach without listing their hostnames
on the Gateway:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: tenant-solar-gateway
namespace: tenant-solar-system
spec:
gatewayClassName: tenants
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: tenant-solar-gateway
allowedListeners:
namespaces:
from: Selector
selector:
matchLabels:
capsule.clastix.io/tenant: solar
listeners:
- name: http-challenge
port: 80
protocol: HTTP
allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespaces:
from: Selector
selector:
matchLabels:
capsule.clastix.io/tenant: solar
allowedListeners controls which namespaces can attach ListenerSets;
allowedRoutes controls which routes can attach to an individual listener.
The Capsule rule supplies the hostname restriction in the tenant namespaces.
The platform-owned Gateway is outside that rule’s scope, so its HTTP challenge
listener can omit hostname.
Tenant listener and route
Assume solar-production belongs to solar, contains a solar-web Service on
port 8080, and has a TLS Secret named solar-web-tls with a certificate for
web.solar.example.com. Alice creates an HTTPS listener in that namespace:
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
name: solar-web
namespace: solar-production
spec:
listeners:
- name: https
hostname: web.solar.example.com
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- group: ""
kind: Secret
name: solar-web-tls
allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespaces:
from: Same
Capsule adds spec.parentRef pointing to tenant-solar-gateway in
tenant-solar-system. The Gateway’s allowedListeners permits the attachment;
the mutation itself does not grant access. The Secret is in the ListenerSet’s
namespace. The HTTPRoute attaches to that ListenerSet, selecting its https listener:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: solar-web
namespace: solar-production
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: ListenerSet
name: solar-web
sectionName: https
hostnames:
- web.solar.example.com
rules:
- backendRefs:
- name: solar-web
port: 8080
Capsule admits both resources because all their hostnames match the Tenant rule.
Changing the ListenerSet hostname or any HTTPRoute hostname to web.lunar.example.com
is denied at admission, even though the Gateway allows ListenerSets from that
namespace. Adding a disallowed hostname alongside web.solar.example.com also
denies the entire request. Omitting a listener hostname or the route’s
spec.hostnames is denied because an allow rule targets these resource types.
1.5 - External Secrets
Integrate shared and per-Tenant secret stores, generate passwords, and distribute credentials with Capsule.
External Secrets Operator (ESO) fetches credentials from a secret backend or generates them, then writes Kubernetes Secrets. Capsule can provision the ESO resources and distribute the resulting Secrets across a Tenant’s namespaces.
This guide covers three patterns:
| Pattern | How it works |
|---|
Shared ClusterSecretStore | Approved Tenants read credentials intended to be shared, using one backend identity. |
Dedicated ClusterSecretStore per Tenant | Capsule generates a store restricted to that Tenant’s namespaces, with a separate backend identity. |
| Generated password per Tenant | ESO creates one source Secret, then Capsule replicates its value into every namespace of that Tenant. |
A namespaced SecretStore is usable in its own namespace. A ClusterSecretStore is cluster-scoped and can be referenced across namespaces, subject to its conditions. Neither resource contains the application credentials itself: an ExternalSecret references the store and describes the target Secret. See ESO’s multi-tenancy guide.
Prerequisites
Run the platform setup as a cluster administrator. You need:
- Capsule with GlobalTenantResource generators and
scope: Tenant support. - ESO installed with the
external-secrets.io/v1 APIs and the generators.external-secrets.io/v1alpha1 Password API. Its controllers must watch the platform and Tenant namespaces and process ClusterSecretStore resources. kubectl, and Python 3 for the password verification example.- For the store examples, a reachable Vault server with a KV v2 engine mounted at
secret, and permission to provision its policies and tokens. Replace https://vault.example.com with your endpoint and configure a trusted CA if needed.
The store examples use Vault to demonstrate backend permissions. The same namespace restrictions apply to other ESO providers, including Azure Key Vault. The password-generation example does not require an external backend or a store.
Example Tenants and namespaces
Save this as tenants.yaml and apply it as a cluster administrator. If these Tenants already exist, add the example label to their current manifests while preserving their owners and other configuration.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
labels:
secrets.example.com/enabled: "true"
spec:
owners:
- kind: Group
name: solar-engineers
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: wind
labels:
secrets.example.com/enabled: "true"
spec:
owners:
- kind: Group
name: wind-engineers
---
apiVersion: v1
kind: Namespace
metadata:
name: solar-dev
labels:
capsule.clastix.io/tenant: solar
---
apiVersion: v1
kind: Namespace
metadata:
name: solar-prod
labels:
capsule.clastix.io/tenant: solar
---
apiVersion: v1
kind: Namespace
metadata:
name: wind-dev
labels:
capsule.clastix.io/tenant: wind
kubectl apply -f tenants.yaml
kubectl get namespaces -L capsule.clastix.io/tenant
The secrets.example.com/enabled label selects Tenants for the GlobalTenantResources below. The capsule.clastix.io/tenant namespace label identifies their ownership and is protected by Capsule. Tenant owners normally create namespaces through Capsule’s namespace workflow.
Keep backend authentication credentials and generated source Secrets in a platform-owned namespace, outside any Tenant. Tenant owners must not have access to this namespace or permission to modify cluster-scoped stores and GlobalTenantResources.
Save this as eso-provisioner.yaml:
apiVersion: v1
kind: Namespace
metadata:
name: capsule-secrets-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: capsule-eso
namespace: capsule-secrets-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: capsule-eso-generators
namespace: capsule-secrets-system
rules:
- apiGroups: ["external-secrets.io"]
resources: ["externalsecrets"]
verbs: ["get", "list", "create", "patch", "delete"]
- apiGroups: ["generators.external-secrets.io"]
resources: ["passwords"]
verbs: ["get", "list", "create", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: capsule-eso-generators
namespace: capsule-secrets-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: capsule-eso-generators
subjects:
- kind: ServiceAccount
name: capsule-eso
namespace: capsule-secrets-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: capsule-eso-replication
rules:
- apiGroups: ["external-secrets.io"]
resources: ["clustersecretstores"]
verbs: ["get", "list", "create", "patch", "delete"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "create", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: capsule-eso-replication
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: capsule-eso-replication
subjects:
- kind: ServiceAccount
name: capsule-eso
namespace: capsule-secrets-system
kubectl apply -f eso-provisioner.yaml
The GlobalTenantResources use this ServiceAccount through Capsule impersonation. The ClusterRole permits Secret replication into current and future Tenant namespaces, which requires broad Secret access; reserve this identity for the platform. ESO uses its own permissions to read authentication credentials, invoke generators, and write source Secrets.
Secure ClusterSecretStores
A store needs two independent access boundaries:
- Kubernetes namespace access:
spec.conditions determines which namespaces may reference the store. - Backend access: the store’s Vault policy, cloud IAM role, or equivalent determines which remote secrets it may fetch.
All users of a store share its backend permissions. A name such as tenant-solar-vault, or a tenant-specific remoteRef.key in an example, does not restrict users to that path. Someone able to create an ExternalSecret can request a different key within the backend identity’s permissions.
For an existing platform-only store, keep its provider configuration and restrict its conditions to the platform namespace:
spec:
conditions:
- namespaces:
- capsule-secrets-system
Alternatively, a namespace selector using matchExpressions with key capsule.clastix.io/tenant and operator DoesNotExist can admit all namespaces outside Capsule Tenants. An explicit namespace allowlist is narrower. With no conditions, a ClusterSecretStore is usable from all namespaces. Conditions are alternatives: matching any entry grants access, so remove broader entries when restricting an existing store. See ClusterSecretStore conditions.
Use a shared ClusterSecretStore
Use a shared store for credentials that all participating Tenants are allowed to read, such as access to a common package registry. This example admits namespaces belonging to solar and wind.
Provision a Vault identity with a policy allowing only the shared KV v2 path. For example, save this as eso-shared.hcl:
path "secret/data/shared/*" {
capabilities = ["read"]
}
The policy uses the Vault KV v2 read path, which includes the data segment. An ESO remoteRef.key below is relative to the secret mount and omits data.
Using an authenticated Vault CLI and jq, create the policy and store its token directly in Kubernetes:
set -o pipefail
vault policy write eso-shared eso-shared.hcl
vault token create -policy=eso-shared -format=json \
| jq -j '.auth.client_token' \
| kubectl -n capsule-secrets-system create secret generic vault-shared \
--from-file=token=/dev/stdin
Manage the token’s expiration, renewal, and rotation as part of your backend setup. ESO also supports Vault Kubernetes authentication when you want to use dedicated ServiceAccount identities instead.
Save this as shared-store.yaml:
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: shared-vault
spec:
conditions:
- namespaceSelector:
matchExpressions:
- key: capsule.clastix.io/tenant
operator: In
values: ["solar", "wind"]
provider:
vault:
server: https://vault.example.com
path: secret
version: v2
auth:
tokenSecretRef:
name: vault-shared
namespace: capsule-secrets-system
key: token
kubectl apply -f shared-store.yaml
kubectl wait --for=condition=Ready --timeout=120s clustersecretstore/shared-vault
Create a Vault record at secret/shared/registry with a password property using your normal secret-management workflow. A user in an admitted namespace can then apply shared-credential.yaml:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: shared-registry
namespace: solar-dev
spec:
refreshPolicy: Periodic
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: shared-vault
target:
name: shared-registry
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: shared/registry
property: password
kubectl apply -f shared-credential.yaml
kubectl -n solar-dev wait --for=condition=Ready --timeout=120s externalsecret/shared-registry
ESO creates the shared-registry Secret in solar-dev. Apply the same ExternalSecret in another admitted namespace to fetch that shared credential there as well. A namespace outside solar and wind cannot use this store, even if its users know the store name.
Create a dedicated ClusterSecretStore per Tenant
For tenant-specific credentials, give each store a different backend identity. This example generates tenant-solar-vault and tenant-wind-vault, with names derived from the Capsule Tenant name.
First provision a separate Vault policy and token for each Tenant. Solar’s eso-tenant-solar.hcl policy is:
path "secret/data/tenants/solar/*" {
capabilities = ["read"]
}
set -o pipefail
vault policy write eso-tenant-solar eso-tenant-solar.hcl
vault token create -policy=eso-tenant-solar -format=json \
| jq -j '.auth.client_token' \
| kubectl -n capsule-secrets-system create secret generic vault-tenant-solar \
--from-file=token=/dev/stdin
Repeat for Wind, using secret/data/tenants/wind/*, policy eso-tenant-wind, and Secret vault-tenant-wind. Do not reuse a token that can read both tenants’ paths. These Kubernetes authentication Secrets hold a token key and remain in the platform namespace. Capsule generates the stores; it does not create Vault policies or tokens.
Save this as tenant-stores.yaml:
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: tenant-secret-stores
spec:
scope: Tenant
resyncPeriod: 60s
serviceAccount:
name: capsule-eso
namespace: capsule-secrets-system
tenantSelector:
matchLabels:
secrets.example.com/enabled: "true"
resources:
- generators:
- missingKey: error
template: |
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: tenant-{{ $.tenant.metadata.name }}-vault
spec:
conditions:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: {{ $.tenant.metadata.name | quote }}
provider:
vault:
server: https://vault.example.com
path: secret
version: v2
auth:
tokenSecretRef:
name: vault-tenant-{{ $.tenant.metadata.name }}
namespace: capsule-secrets-system
key: token
kubectl apply -f tenant-stores.yaml
kubectl wait --for=condition=Ready --timeout=120s globaltenantresource/tenant-secret-stores
kubectl wait --for=condition=Ready --timeout=120s \
clustersecretstore/tenant-solar-vault clustersecretstore/tenant-wind-vault
scope: Tenant creates one cluster-scoped store per Tenant. The store itself has no metadata.namespace; its authentication Secret reference must include one. Its namespace selector automatically admits new namespaces owned by that Tenant.
After creating secret/tenants/solar/application in Vault with a password property, apply tenant-credential.yaml:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: application-credentials
namespace: solar-dev
spec:
refreshPolicy: Periodic
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: tenant-solar-vault
target:
name: application-credentials
creationPolicy: Owner
data:
- secretKey: password
remoteRef:
key: tenants/solar/application
property: password
kubectl apply -f tenant-credential.yaml
kubectl -n solar-dev wait --for=condition=Ready --timeout=120s \
externalsecret/application-credentials
Check both boundaries when validating this setup: an ExternalSecret in wind-dev referencing tenant-solar-vault should be denied by the store’s namespace conditions. An ExternalSecret in solar-dev using its own store but requesting tenants/wind/application should be denied by Vault. Use a separate test resource and target Secret when checking denied requests, and inspect its conditions with kubectl describe externalsecret.
Generate one password and distribute it to every Tenant namespace
An ESO Password generator produces a new random value each time it is invoked. Referencing the same generator from several ExternalSecrets produces different passwords. To share one credential throughout a Tenant, generate it in one source Secret and replicate that Secret. The generator resource holds the password-generation settings, not the generated password. See generator behavior.
flowchart LR
GTR[GlobalTenantResource: scope Tenant] --> Password[Password generator]
GTR --> ES[One ExternalSecret per Tenant]
Password --> ES
ES --> Source[Source Secret in capsule-secrets-system]
Source --> Replication[GlobalTenantResource: scope Namespace]
Replication --> Dev[Same Secret in solar-dev]
Replication --> Prod[Same Secret in solar-prod]
Generate the source Secrets
Save this as tenant-password-sources.yaml. It creates a Password generator and an ExternalSecret for each enabled Tenant, all in capsule-secrets-system:
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: tenant-password-sources
spec:
scope: Tenant
resyncPeriod: 60s
serviceAccount:
name: capsule-eso
namespace: capsule-secrets-system
tenantSelector:
matchLabels:
secrets.example.com/enabled: "true"
resources:
- generators:
- missingKey: error
template: |
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
name: tenant-{{ $.tenant.metadata.name }}-password
namespace: capsule-secrets-system
spec:
length: 32
digits: 6
symbols: 0
noUpper: false
allowRepeat: true
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: tenant-{{ $.tenant.metadata.name }}-password
namespace: capsule-secrets-system
spec:
refreshPolicy: CreatedOnce
target:
name: tenant-{{ $.tenant.metadata.name }}-password
creationPolicy: Owner
template:
engineVersion: v2
mergePolicy: Merge
type: Opaque
metadata:
labels:
secrets.example.com/tenant: {{ $.tenant.metadata.name | quote }}
secrets.example.com/purpose: shared-password
dataFrom:
- sourceRef:
generatorRef:
apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
name: tenant-{{ $.tenant.metadata.name }}-password
kubectl apply -f tenant-password-sources.yaml
kubectl wait --for=condition=Ready --timeout=120s globaltenantresource/tenant-password-sources
kubectl -n capsule-secrets-system wait --for=condition=Ready --timeout=120s \
externalsecret/tenant-solar-password externalsecret/tenant-wind-password
For Solar, ESO creates capsule-secrets-system/tenant-solar-password with a password data key. Wind gets an independent value in tenant-wind-password. A generator reference resolves in the ExternalSecret’s namespace and does not need secretStoreRef.
CreatedOnce avoids scheduled regeneration. Capsule’s resyncPeriod reconciles the ESO resource definitions; it does not request a new password on every pass. However, ESO can sync again if the target Secret changes or is deleted, or if the ExternalSecret is recreated. Treat source recreation as credential rotation and retain a backup if the credential must survive recovery.
The explicit target.template.metadata is significant: it defines the source Secret’s labels instead of inheriting the Capsule management labels from its ExternalSecret. Capsule skips Secrets marked projectcapsule.dev/created-by: resources during replication to prevent loops. Keep the source Secret owned by ESO and its replicas owned by Capsule; do not stamp Capsule’s reserved labels onto the source. ESO’s template metadata behavior controls this distinction.
Replicate the source into all namespaces
Save this as tenant-password-replication.yaml:
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: tenant-password-replication
spec:
scope: Namespace
resyncPeriod: 60s
serviceAccount:
name: capsule-eso
namespace: capsule-secrets-system
tenantSelector:
matchLabels:
secrets.example.com/enabled: "true"
resources:
- namespacedItems:
- apiVersion: v1
kind: Secret
namespace: capsule-secrets-system
name: "tenant-{{tenant.name}}-password"
optional: false
kubectl apply -f tenant-password-replication.yaml
kubectl wait --for=condition=Ready --timeout=120s globaltenantresource/tenant-password-replication
The namespaced reference selects one source by name for each Tenant. With scope: Namespace and no namespaceSelector, Capsule copies it into every namespace of that Tenant, preserving its name and data. New namespaces receive the existing value on reconciliation. optional: false reports a missing source instead of silently producing no copies.
Capsule’s generator and replication readiness only covers Kubernetes resources. It does not prove that ESO has created the source Secret, which is why the commands wait for the source ExternalSecrets separately. If the two GlobalTenantResources are applied together, replication may initially report a missing Secret and then succeed once ESO creates it.
Verify identical values and tenant isolation
Check that both Solar namespaces have Solar’s Secret and Wind has only its own:
kubectl -n solar-dev get secret tenant-solar-password
kubectl -n solar-prod get secret tenant-solar-password
kubectl -n wind-dev get secret tenant-wind-password
kubectl -n wind-dev get secret tenant-solar-password
The last command should report NotFound. This Python check compares the values without printing passwords or hashes:
import base64
import json
import subprocess
def password(namespace, name):
raw = subprocess.check_output(
["kubectl", "-n", namespace, "get", "secret", name, "-o", "json"]
)
return base64.b64decode(json.loads(raw)["data"]["password"])
solar = password("capsule-secrets-system", "tenant-solar-password")
wind = password("capsule-secrets-system", "tenant-wind-password")
assert len(solar) == 32
assert solar == password("solar-dev", "tenant-solar-password")
assert solar == password("solar-prod", "tenant-solar-password")
assert wind == password("wind-dev", "tenant-wind-password")
assert solar != wind
print("Passwords match within each Tenant and differ between Tenants.")
Create another namespace belonging to Solar and repeat the comparison after replication. The new namespace should receive the same source value; adding a namespace does not invoke the Password generator again.
Workloads consume the local copy through a Secret volume or secretKeyRef. For example, use this fragment in a Pod template in solar-dev or solar-prod:
env:
- name: APPLICATION_PASSWORD
valueFrom:
secretKeyRef:
name: tenant-solar-password
key: password
Register the generated credential with the service that authenticates it. Creating or copying a Secret does not set a database user’s password or otherwise configure that service.
Rotation and lifecycle
For backend-managed values, update the backend record. The Periodic ExternalSecrets fetch it on their next refresh. Backend authentication tokens have their own lifecycle and must be renewed or replaced separately.
For generated passwords, CreatedOnce keeps the source stable during ordinary reconciliation. Recreating the source can produce a different password, and Capsule will then distribute the new value. Coordinate rotation with the authenticating service and all consuming workloads. Applications using environment variables need a restart to load changed Secret values; applications reading Secret volumes need to reload them.
The example uses creationPolicy: Owner, so deleting the source ExternalSecret also makes its source Secret eligible for garbage collection. Removing the Tenant’s opt-in label or deleting its GlobalTenantResources can prune generated objects and replicas under Capsule’s object management rules. Check that replicas are removed and service credentials are revoked during offboarding. Source removal alone is not immediate credential revocation.
For credentials that must survive deleting and recreating an ExternalSecret, ESO documents creationPolicy: Orphan with an immutable target. That changes cleanup and rotation: retained source Secrets need explicit offboarding, and immutable replicas cannot be updated in place. Choose that lifecycle deliberately rather than adding immutability to the replication example unchanged.
1.6 - FluxCD
Give each Tenant a Flux reconciliation identity and let its owners view, reconcile, and suspend their deployments.
Flux can reconcile each Tenant’s Git repository using a dedicated ServiceAccount. Capsule makes that account a Tenant owner and distributes its permissions to the Tenant’s namespaces. Tenant owners can see their Flux resources through Capsule Proxy and request reconciliation or suspend deployment.
The example uses a shared, platform-owned tenant-system namespace outside every Tenant. Solar gets tenant-system/flux-solar; Wind gets a different account. Neither Tenant administers this namespace.
flowchart LR
GTR[GlobalTenantResource] --> SA[ServiceAccount in tenant-system]
GTR --> TO[TenantOwner]
TO --> Tenant[Tenant ownership]
Rules[Tenant permission rules] --> Namespaces[Admin RoleBindings in Tenant namespaces]
SA --> Namespaces
Repo[GitRepository] --> KS[Kustomization in tenant-system]
KS -->|impersonates| SA
Owners[Tenant owners] -->|view through Capsule Proxy| Repo
Owners -->|reconcile or suspend| KS
Prerequisites
Run platform setup as a cluster administrator. You need:
- Capsule with
TenantOwner, Tenant permission rules, and GlobalTenantResource generators with scope: Tenant. - Capsule Proxy configured in the kubeconfig files used by Tenant users. Install Flux CRDs before starting the proxy so discovery includes them; restart the proxy after adding these APIs if needed.
- Flux watching
tenant-system, with source.toolkit.fluxcd.io/v1 GitRepository and kustomize.toolkit.fluxcd.io/v1 Kustomization APIs. The Kustomize controller must be allowed to impersonate the reconciliation accounts. - Flux’s standard RBAC aggregation into
admin, or equivalent list permissions for Flux resources in every Tenant namespace. Capsule Proxy uses those permissions to authorize the Tenant’s filtered collection view. - Kubernetes 1.30 or later with
admissionregistration.k8s.io/v1 ValidatingAdmissionPolicy enabled for the delegation guard below. kubectl, and optionally the Flux CLI.
Run Flux safely in a shared cluster
Before connecting Tenant repositories, follow Flux’s official multi-tenancy lockdown guide. Use its controller configuration for your installed Flux version. Capsule’s namespace ownership and permission rules work alongside Flux’s own authorization controls.
The upstream guide covers the settings needed for these examples:
- Disable cross-namespace Flux references with
--no-cross-namespace-refs=true on the relevant controllers. - Disable Kustomize remote bases with
--no-remote-bases=true. - Configure
--default-service-account=default for the Kustomize and Helm controllers, and leave that fallback account without reconciliation privileges. Each Tenant Kustomization or HelmRelease must explicitly select its authorized account with spec.serviceAccountName. - Keep privileged platform reconciliation and its source repositories under platform administrator control.
These restrictions concern Flux source and object references across namespaces; they do not prevent an authorized reconciliation account from deploying into several namespaces of its Capsule Tenant. Both integration patterns below keep each Flux object and its source in the same namespace.
Several Tenants share tenant-system in the first pattern. Flux’s namespace reference restriction does not isolate objects within that shared namespace: keep their definitions platform-managed and apply the delegation guard before granting owners patch access. In the HelmRelease pattern, the source, release, and promoted account live inside the Tenant instead.
Refer to Flux’s security best practices for the wider deployment configuration and its multi-tenancy example repository for platform and Tenant repository layouts.
Save this as flux-platform.yaml. The capsule-flux-bootstrap account runs the GlobalTenantResource; it is separate from the accounts Flux impersonates when applying Tenant workloads.
apiVersion: v1
kind: Namespace
metadata:
name: tenant-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: capsule-flux-bootstrap
namespace: tenant-system
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: capsule-flux-bootstrap
namespace: tenant-system
rules:
- apiGroups: [""]
resources: ["serviceaccounts"]
verbs: ["get", "list", "create", "patch", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["get", "list", "create", "update", "patch", "delete"]
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles"]
verbs: ["bind", "escalate"]
- apiGroups: ["source.toolkit.fluxcd.io"]
resources: ["gitrepositories"]
verbs: ["get", "list", "create", "patch", "delete"]
- apiGroups: ["kustomize.toolkit.fluxcd.io"]
resources: ["kustomizations"]
verbs: ["get", "list", "create", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: capsule-flux-bootstrap
namespace: tenant-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: capsule-flux-bootstrap
subjects:
- kind: ServiceAccount
name: capsule-flux-bootstrap
namespace: tenant-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: capsule-flux-bootstrap
rules:
- apiGroups: ["capsule.clastix.io"]
resources: ["tenantowners"]
verbs: ["get", "list", "create", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: capsule-flux-bootstrap
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: capsule-flux-bootstrap
subjects:
- kind: ServiceAccount
name: capsule-flux-bootstrap
namespace: tenant-system
kubectl apply -f flux-platform.yaml
bind and escalate allow this platform identity to generate the Roles and RoleBindings below. Creating TenantOwners also grants ownership, so keep this identity and its RBAC under administrator control. Tenant owners receive no access to ServiceAccounts, tokens, Secrets, or RBAC in tenant-system.
Grant the reconciliation account access to Tenant namespaces
Save this as solar-tenant.yaml. For an existing Tenant, merge the label and rule into its current definition, preserving its owners and other rules.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
labels:
gitops.example.com/enabled: "true"
spec:
owners:
- kind: User
name: alice
rules:
- permissions:
bindings:
- clusterRoleName: admin
subjects:
- kind: ServiceAccount
name: flux-{{ .tenant.metadata.name }}
namespace: tenant-system
kubectl apply -f solar-tenant.yaml
With no namespaceSelector, Capsule creates the admin RoleBinding in every current and future namespace belonging to Solar. This is namespaced administration, not a cluster-admin grant. Add namespace selectors when different namespace profiles need different access.
Use permissions.bindings for this external account: permissions.promotions discovers labelled ServiceAccounts inside the selected Tenant namespaces and will not discover an account in tenant-system. Ownership is established separately by the TenantOwner below. See Tenant permission rules.
Generate identities and owner permissions
Save this as flux-tenants.yaml. One GlobalTenantResource creates an account, a TenantOwner, and a named Role and RoleBinding per enabled Tenant. It runs once per Tenant, even when the Tenant has no namespaces yet.
Before applying this GlobalTenantResource, install the delegation guard.
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: flux-tenants
spec:
scope: Tenant
resyncPeriod: 60s
serviceAccount:
name: capsule-flux-bootstrap
namespace: tenant-system
tenantSelector:
matchLabels:
gitops.example.com/enabled: "true"
resources:
- policy:
creation: Owner
protect: true
deletion: Remove
generators:
- missingKey: error
template: |
apiVersion: v1
kind: ServiceAccount
metadata:
name: flux-{{ $.tenant.metadata.name }}
namespace: tenant-system
labels:
projectcapsule.dev/tenant: {{ $.tenant.metadata.name | quote }}
automountServiceAccountToken: false
---
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
name: flux-{{ $.tenant.metadata.name }}
labels:
projectcapsule.dev/tenant: {{ $.tenant.metadata.name | quote }}
spec:
kind: ServiceAccount
name: system:serviceaccount:tenant-system:flux-{{ $.tenant.metadata.name }}
clusterRoles: []
aggregate: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: flux-{{ $.tenant.metadata.name }}-owners
namespace: tenant-system
rules:
- apiGroups: ["source.toolkit.fluxcd.io"]
resources: ["gitrepositories"]
resourceNames: ["flux-{{ $.tenant.metadata.name }}"]
verbs: ["get"]
- apiGroups: ["kustomize.toolkit.fluxcd.io"]
resources: ["kustomizations"]
resourceNames: ["flux-{{ $.tenant.metadata.name }}"]
verbs: ["get", "patch"]
{{- $owners := $.tenant | get "status" | default (dict) | get "owners" | default (list) }}
{{- if $owners }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: flux-{{ $.tenant.metadata.name }}-owners
namespace: tenant-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: flux-{{ $.tenant.metadata.name }}-owners
subjects:
{{- range $owners }}
{{- if eq .kind "ServiceAccount" }}
- kind: ServiceAccount
name: {{ index (splitList ":" .name) 3 | quote }}
namespace: {{ index (splitList ":" .name) 2 | quote }}
{{- else }}
- kind: {{ .kind }}
apiGroup: rbac.authorization.k8s.io
name: {{ .name | quote }}
{{- end }}
{{- end }}
{{- end }}
kubectl apply -f flux-tenants.yaml
kubectl wait --for=condition=Ready --timeout=120s globaltenantresource/flux-tenants
kubectl wait --for=condition=Ready --timeout=120s tenantowner/flux-solar
kubectl get tenant solar -o jsonpath='{.status.owners}'
kubectl -n tenant-system get rolebinding flux-solar-owners -o yaml
The projectcapsule.dev/tenant: solar label assigns the TenantOwner to Solar without a matchOwners selector. aggregate: true also registers the account as a Capsule user. Its empty clusterRoles list leaves workload permissions to the Tenant rule rather than the default owner roles.
The generated RoleBinding follows effective tenant.status.owners, including users, groups, and ServiceAccounts. It refreshes as the replication reconciles; wait until the subjects reflect the intended owners before testing access or considering a revocation complete. A ServiceAccount owner uses its Kubernetes name and namespace in the RoleBinding, rather than its full authentication username.
Connect a Tenant repository
Create the Flux resources manually as a platform administrator. Save this as solar-flux.yaml, replacing the URL, branch, and path with the Tenant’s repository settings. The path must contain a Kustomize configuration whose namespaced manifests target Solar namespaces that already exist. Omit targetNamespace to allow manifests in several Solar namespaces.
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-solar
namespace: tenant-system
labels:
projectcapsule.dev/tenant: solar
capsule.clastix.io/managed-by: solar
spec:
interval: 1m
url: https://github.com/example/solar-gitops.git
ref:
branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-solar
namespace: tenant-system
labels:
projectcapsule.dev/tenant: solar
capsule.clastix.io/managed-by: solar
spec:
interval: 5m
path: ./clusters/production
prune: true
serviceAccountName: flux-solar
sourceRef:
kind: GitRepository
name: flux-solar
kubectl apply -f solar-flux.yaml
kubectl -n tenant-system wait --for=condition=Ready --timeout=120s gitrepository/flux-solar
kubectl -n tenant-system wait --for=condition=Ready --timeout=120s kustomization/flux-solar
Flux impersonates tenant-system/flux-solar to apply the repository’s workloads. The account has namespace administration only where Capsule has bound it; manifests targeting another Tenant’s namespaces must fail authorization. For private repositories, provision a separate read-only Git credential Secret in tenant-system and reference it with GitRepository.spec.secretRef.name. Keep credentials under platform control. See the Flux GitRepository documentation.
The labels have different purposes:
| Label | Purpose |
|---|
projectcapsule.dev/tenant: solar | Assigns the TenantOwner and identifies the Tenant on generated resources. |
capsule.clastix.io/managed-by: solar | Selects the Flux resources for Capsule Proxy’s generic namespaced collection filtering. Keep this compatibility label on both Flux resources for that proxy path. |
Labels do not grant Kubernetes RBAC permissions. The proxy’s filtered collection view and the named Role grants work together: users list their resources through the proxy, then use explicit get and patch permissions on their own named objects. Do not grant list on the whole shared namespace or substitute a GlobalProxySettings.clusterResources rule, which is for cluster-scoped resources.
View, reconcile, and suspend
Run these commands with Alice’s kubeconfig pointing to Capsule Proxy:
kubectl get gitrepositories.source.toolkit.fluxcd.io --all-namespaces
kubectl get kustomizations.kustomize.toolkit.fluxcd.io --all-namespaces
kubectl -n tenant-system get kustomization flux-solar
flux reconcile kustomization flux-solar --namespace tenant-system
flux suspend kustomization flux-solar --namespace tenant-system
flux resume kustomization flux-solar --namespace tenant-system
The reconcile command uses the current source artifact. --with-source additionally patches the GitRepository and is intentionally outside the read-only source permission in this example. Request a source refresh through the platform or wait for its interval. Reconciliation requests set reconcile.fluxcd.io/requestedAt; suspend and resume change spec.suspend. See Flux’s reconciliation and suspension examples.
Verify the persisted spec.suspend value after suspending and resuming. Suspension pauses future reconciliation; it does not remove deployed workloads.
Repeat setup for a second Tenant such as Wind with a different owner and repository. Verify that:
- Alice’s collection results include Solar’s Flux resources and exclude Wind’s.
- Alice can reconcile, suspend, and resume
flux-solar, with status reflecting the requested operation. - Alice cannot get or patch
tenant-system/flux-wind and cannot read Secrets or mint ServiceAccount tokens in tenant-system. - An attempted patch of
flux-solar.spec.serviceAccountName, sourceRef, or its Tenant labels is denied by capsule-flux-delegation, with the stored object unchanged. - A Solar repository change targeting a Wind namespace fails with the expected authorization error and leaves Wind unchanged.
Automate the Flux definitions too
For new Tenants, the same GlobalTenantResource can also generate the GitRepository and Kustomization. Store their repository settings in Tenant data:
spec:
data:
flux:
url: https://github.com/example/solar-gitops.git
branch: main
path: ./clusters/production
Append this block to spec.resources in flux-tenants.yaml before adding those Tenants:
- policy:
creation: Owner
protect: false
deletion: Remove
generators:
- missingKey: error
template: |
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: flux-{{ $.tenant.metadata.name }}
namespace: tenant-system
labels:
projectcapsule.dev/tenant: {{ $.tenant.metadata.name | quote }}
capsule.clastix.io/managed-by: {{ $.tenant.metadata.name | quote }}
spec:
interval: 1m
url: {{ $.tenant.spec.data.flux.url | quote }}
ref:
branch: {{ $.tenant.spec.data.flux.branch | quote }}
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: flux-{{ $.tenant.metadata.name }}
namespace: tenant-system
labels:
projectcapsule.dev/tenant: {{ $.tenant.metadata.name | quote }}
capsule.clastix.io/managed-by: {{ $.tenant.metadata.name | quote }}
spec:
interval: 5m
path: {{ $.tenant.spec.data.flux.path | quote }}
prune: true
serviceAccountName: flux-{{ $.tenant.metadata.name }}
sourceRef:
kind: GitRepository
name: flux-{{ $.tenant.metadata.name }}
Provide spec.data.flux for every Tenant selected by the replication before adding this block. creation: Owner expects absent objects or objects already managed by this replication; adopting manually created definitions requires a deliberate Merge policy.
protect: false allows the delegated Kustomization patches; the admission guard still limits their fields. Keep spec.suspend and the reconcile request annotation out of the generator so replication does not continually reset the owner’s actions. The identity and RBAC block remains protected. Bootstrap and repository settings stay platform-managed, while namespace permission rules continue to define where each reconciliation account can act.
Deleting the replication or deselecting a Tenant removes its generated objects with deletion: Remove. When Flux definitions are generated too, deleting a Kustomization with prune: true can delete its workloads. Review Flux’s deletion behavior before removing a Tenant from this integration; manually created Flux definitions require separate cleanup. Remove the Tenant’s reconciliation-account binding rule when retiring the integration.
Let a Tenant HelmRelease create namespaces
Tenant owners can also run their own HelmRelease inside a Tenant namespace. When its chart creates additional namespaces, the Helm reconciliation account needs Tenant ownership as well as permissions to apply namespaced resources. A namespace admin RoleBinding alone does not authorize namespace creation.
Apply the Flux multi-tenancy configuration to the Helm controller too. Owner promotion grants the selected account Tenant access; it does not configure the controller’s fallback identity or restrictions on source references.
For this pattern, create the account in solar-system, a namespace belonging to Solar. This differs from the platform-owned tenant-system namespace used earlier: label-based owner promotion discovers accounts inside the Tenant’s namespaces. No separate TenantOwner object is needed for this account.
Route Helm’s API requests through Capsule Proxy
Configure the Helm controller’s Kubernetes client to use Capsule Proxy. Helm checks whether a chart’s resources already exist before creating them. A direct API request for a missing namespace can return Forbidden before Capsule has created its RoleBindings; the proxy provides the namespace lookup behavior needed here. Configuring Alice’s kubeconfig alone does not change the controller’s connection.
For a controller using the in-cluster client, the following deployment patch redirects it to capsule-proxy.capsule-system.svc:9001. Adjust the endpoint for your proxy installation. As a platform administrator, first save the trusted proxy CA certificate as capsule-proxy-ca.crt and create its ConfigMap:
kubectl -n flux-system create configmap capsule-proxy-ca \
--from-file=ca.crt=./capsule-proxy-ca.crt \
--dry-run=client -o yaml | kubectl apply -f -
Save this patch as helm-controller-proxy.yaml and include it in the platform’s Flux installation configuration so Git reconciliation preserves it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: helm-controller
namespace: flux-system
spec:
template:
spec:
containers:
- name: manager
env:
- name: KUBERNETES_SERVICE_HOST
value: capsule-proxy.capsule-system.svc
- name: KUBERNETES_SERVICE_PORT
value: "9001"
volumeMounts:
- name: capsule-proxy-ca
mountPath: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
subPath: ca.crt
readOnly: true
volumes:
- name: capsule-proxy-ca
configMap:
name: capsule-proxy-ca
For an immediate application of that same patch:
kubectl -n flux-system patch deployment helm-controller \
--type=strategic --patch-file helm-controller-proxy.yaml
kubectl -n flux-system rollout status deployment/helm-controller
The projected ServiceAccount token remains in place; only its CA file is overlaid. Restart the controller after updating this CA ConfigMap because a subPath mount does not receive ConfigMap updates. The proxy must trust the controller’s authentication and permit its authorized impersonation requests.
As a platform administrator, enable ServiceAccount promotion in the existing CapsuleConfiguration, preserving its other settings. Replace default if your configuration has another name:
kubectl patch capsuleconfiguration default --type=merge \
--patch '{"spec":{"allowServiceAccountPromotion":true}}'
Merge the following into Solar’s Tenant definition, preserving its owners and existing rules:
spec:
permissions:
allowOwnerPromotion: true
rules:
- permissions:
promotions:
- clusterRoles: ["admin"]
selector:
matchLabels:
app.kubernetes.io/name: helm-reconciler
This rule grants matching, promoted accounts admin in every current and future Solar namespace. Add a namespaceSelector if only selected namespace profiles should receive that access; the namespace containing the account must also match for rule-based discovery. The ClusterRoles configured in CapsuleConfiguration.spec.rbac.promotionClusterRoles apply independently to owner promotion. Keep the namespace provisioning permissions required by your Capsule installation. See owner promotion and rule-based promotion.
Ensure solar-system exists and belongs to Solar before creating the account. For example, a platform administrator can apply:
apiVersion: v1
kind: Namespace
metadata:
name: solar-system
labels:
capsule.clastix.io/tenant: solar
As Alice, or another existing Solar owner, apply solar-helm-account.yaml:
apiVersion: v1
kind: ServiceAccount
metadata:
name: helm-reconciler
namespace: solar-system
labels:
app.kubernetes.io/name: helm-reconciler
owner.projectcapsule.dev/promote: "true"
projectcapsule.dev/promote: "true"
automountServiceAccountToken: false
kubectl apply -f solar-helm-account.yaml
kubectl get tenant solar -o jsonpath='{.status.owners}'
kubectl get tenant solar -o jsonpath='{.status.promotions}'
kubectl -n solar-system get rolebindings -o yaml
The two promotion labels have separate purposes:
| Label | Effect |
|---|
owner.projectcapsule.dev/promote: "true" | Makes the account an owner of the Tenant containing its namespace, allowing Tenant namespace creation under Capsule’s admission policies. |
projectcapsule.dev/promote: "true" | Makes the account eligible for the Tenant rule above, which distributes namespaced admin permissions. |
Wait until system:serviceaccount:solar-system:helm-reconciler appears in Solar’s effective owners and promotions, and the corresponding admin RoleBinding exists in solar-system, before creating the HelmRelease. The Helm controller must be allowed to impersonate the account. Promotion must be performed by an existing authorized owner; do not rely on the account promoting itself during chart installation.
Put namespace definitions in the chart
In the Tenant’s Git repository, create charts/tenant-namespaces/Chart.yaml:
apiVersion: v2
name: tenant-namespaces
description: Namespaces managed by the Solar Tenant
type: application
version: 0.1.0
Create charts/tenant-namespaces/templates/namespaces.yaml:
{{- range .Values.namespaces }}
apiVersion: v1
kind: Namespace
metadata:
name: {{ . | quote }}
labels:
capsule.clastix.io/tenant: {{ $.Values.tenant | quote }}
annotations:
helm.sh/resource-policy: keep
---
{{- end }}
The explicit Tenant label supports Capsule’s label-based namespace assignment. Prefix, namespace quota, and other namespace policies still apply. The keep annotation retains these namespaces when the Helm release is uninstalled; clean them up deliberately when they are no longer needed.
This example chart creates only namespaces. Capsule reconciles their namespace profiles and RoleBindings asynchronously, so wait for those permissions before deploying workloads into a newly created namespace.
Save this as solar-helm.yaml, replacing the repository URL. Keep the source in the same namespace as the HelmRelease to work with Flux’s cross-namespace reference restrictions:
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: tenant-workloads
namespace: solar-system
spec:
interval: 1m
url: https://github.com/example/solar-gitops.git
ref:
branch: main
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: tenant-namespaces
namespace: solar-system
spec:
interval: 5m
serviceAccountName: helm-reconciler
targetNamespace: solar-system
storageNamespace: solar-system
chart:
spec:
chart: ./charts/tenant-namespaces
interval: 1m
sourceRef:
kind: GitRepository
name: tenant-workloads
values:
tenant: solar
namespaces:
- solar-preview
kubectl apply -f solar-helm.yaml
kubectl -n solar-system wait --for=condition=Ready --timeout=120s helmrelease/tenant-namespaces
kubectl get namespace solar-preview -o yaml
kubectl -n solar-preview get rolebindings -o yaml
Confirm that the persisted namespace belongs to Solar and receives the expected profile and admin binding for helm-reconciler. Changing the chart to claim a namespace for another Tenant must be rejected by Capsule. Owner promotion does not grant cluster-admin or permission to install arbitrary cluster-scoped resources such as CRDs and ClusterRoles.
Flux also supports spec.install.createNamespace: true for a missing target namespace. That option synthesizes a namespace without the explicit Tenant label shown above, so it requires a compatible Capsule namespace assignment configuration, such as enforced Tenant prefixes. Use chart-defined, labelled namespaces for the label-based assignment pattern in this guide. See the Flux HelmRelease reference.
To revoke both promotions, remove both labels and update any Git manifest that would restore them:
kubectl -n solar-system label serviceaccount helm-reconciler \
owner.projectcapsule.dev/promote- projectcapsule.dev/promote-
Wait until the account disappears from Solar’s effective owners and promotions and the generated grants are removed. Revocation does not delete namespaces or deployed resources, and subsequent Helm reconciliation can fail until an authorized replacement account is configured.
Limit delegated changes to reconcile and suspend
Kubernetes RBAC cannot grant permission to patch just one field. Without admission validation, permission to suspend a Kustomization also permits changing spec.serviceAccountName to another Tenant’s account in the shared namespace.
Save this as flux-delegation-guard.yaml and apply it before granting Tenant owners patch access. It permits changes to spec.suspend and the reconciliation request annotation while preserving the rest of the specification and the ownership metadata. Actors allowed to update the platform bootstrap Role are exempt so administrators and the bootstrap identity can manage the definitions. Never grant Tenant owners that Role permission. The Flux controller is also exempt so it can manage finalizers; adjust its ServiceAccount identity if Flux runs outside flux-system.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: capsule-flux-delegation
spec:
failurePolicy: Fail
matchConstraints:
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: tenant-system
resourceRules:
- apiGroups: ["kustomize.toolkit.fluxcd.io"]
apiVersions: ["v1"]
operations: ["UPDATE"]
resources: ["kustomizations"]
matchConditions:
- name: not-flux-controller
expression: "request.userInfo.username != 'system:serviceaccount:flux-system:kustomize-controller'"
- name: delegated-user
expression: >-
!authorizer.group('rbac.authorization.k8s.io').resource('roles')
.namespace('tenant-system').name('capsule-flux-bootstrap')
.check('update').allowed()
variables:
- name: annotations
expression: "has(object.metadata.annotations) ? object.metadata.annotations : {}"
- name: oldAnnotations
expression: "has(oldObject.metadata.annotations) ? oldObject.metadata.annotations : {}"
validations:
- expression: >-
object.spec.all(k, k == 'suspend' ||
(k in oldObject.spec && object.spec[k] == oldObject.spec[k])) &&
oldObject.spec.all(k, k == 'suspend' ||
(k in object.spec && oldObject.spec[k] == object.spec[k]))
message: "Tenant owners may change only spec.suspend."
- expression: >-
object.metadata.?labels.orValue({}) == oldObject.metadata.?labels.orValue({}) &&
object.metadata.?ownerReferences.orValue([]) == oldObject.metadata.?ownerReferences.orValue([]) &&
object.metadata.?finalizers.orValue([]) == oldObject.metadata.?finalizers.orValue([])
message: "Tenant owners must preserve labels, owner references, and finalizers."
- expression: >-
variables.annotations.all(k, k == 'reconcile.fluxcd.io/requestedAt' ||
(k in variables.oldAnnotations && variables.annotations[k] == variables.oldAnnotations[k])) &&
variables.oldAnnotations.all(k, k == 'reconcile.fluxcd.io/requestedAt' ||
(k in variables.annotations && variables.oldAnnotations[k] == variables.annotations[k]))
message: "Tenant owners may change only the reconciliation request annotation."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: capsule-flux-delegation
spec:
policyName: capsule-flux-delegation
validationActions: ["Deny"]
kubectl apply -f flux-delegation-guard.yaml
The policy excludes the status subresource, so Flux can report reconciliation results. See Kubernetes’ ValidatingAdmissionPolicy reference.
1.7 - Headlamp
Capsule Integration with Headlamp
Headlamp is an easy-to-use and extensible Kubernetes web UI.
Headlamp was created to blend the traditional feature set of other web UIs/dashboards (i.e., to list and view resources) with added functionality.
Prerequisites
- You will need a running Capsule Proxy instance.
- For Authentication you will need a Confidential OIDC client configured in your OIDC provider, such as Keycloak, Dex, or Google Cloud Identity. By default the Kubernetes API only validates tokens against a Public OIDC client, so you will need to configure your OIDC provider to allow the Headlamp client to issue tokens. You must make use of the Kubernetes Authentication Configuration, which allows to define multiple audiences (clients). This way we can issue tokens for a headlamp client, which is Confidential (Client Secret), and a kubernetes client, which is Public. The Kubernetes API will validate the tokens against both clients. The Config might look like this:
apiVersion: apiserver.config.k8s.io/v1beta1
kind: AuthenticationConfiguration
jwt:
- issuer:
url: https://keycloak/realms/realm-name
audiences:
- kubernetes
- headlamp
audienceMatchPolicy: MatchAny # This one is important
claimMappings:
username:
claim: 'email'
prefix: ""
groups:
claim: 'groups'
prefix: ""
Read More
Integration
To install Headlamp, you can use the Helm chart provided in the Headlamp repository. It’s a bit special how Headlamp handles Certificate Authorities. We need to inject the capsule-proxy CA into the trust store for Headlamp. In the below example we are using the CA Bundle from alpine (because we also need to trust the CA of the OIDC Issuer, in this case it uses Let’s Encrypt). See the issues #3707 and #127. Essentially Golang uses some environment variables to allow specifying cert files/dirs overriding the system defaults. By Default this is under /etc/ssl/. You can change this behavior by defining the environment variables SSL_CERT_FILE or/and SSL_CERT_DIR.
It’s recommended to install headlamp in the capsule-system namespace. Otherwise you need to somehow replicate the internal ca secret to the namespace, where headlamp is deployed to. For this case Cert-Manager Trust-Bundles might be useful.
With the following values we got it to work:
config:
inCluster: true
extraArgs:
- -insecure-ssl
env:
- name: KUBERNETES_SERVICE_HOST
value: "capsule-proxy.capsule-system.svc"
- name: KUBERNETES_SERVICE_PORT
value: "9001"
- name: "OIDC_ISSUER_URL"
value: "https://keycloak/realms/realm-name"
- name: "OIDC_CLIENT_ID"
value: "headlamp"
- name: "OIDC_CLIENT_SECRET"
value: "<SECRET>"
- name: "OIDC_USE_ACCESS_TOKEN"
value: "false"
#- name: "OIDC_SCOPES"
# value: "openid profile email groups offline_access"
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: token-ca
- name: ca-store
mountPath: /etc/ssl/
volumes:
- name: ca-store
emptyDir: {}
- name: capsule-proxy
secret:
secretName: capsule-proxy
- name: token-ca
projected:
sources:
- serviceAccountToken:
path: token
- secret:
items:
- key: ca.crt
path: ca.crt
name: capsule-proxy
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
initContainers:
- name: add-ca
image: alpine:3
command: ["/bin/sh","-c"]
args:
- |
set -e
cp -R /etc/ssl/* /work/
cat /ca/ca.crt >> /work/certs/ca-certificates.crt
volumeMounts:
- name: ca-store
mountPath: /work
- name: capsule-proxy
mountPath: /ca
securityContext:
runAsNonRoot: true
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
privileged: false
runAsUser: 65534
runAsGroup: 65534
podSecurityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
privileged: false
runAsUser: 100
runAsGroup: 101
fsGroup: 101
fsGroupChangePolicy: "Always"
Note: The secret capsule-proxy refers to the secret which is being used by the capsule-proxy instance directly, not the self-signed-ca secret.
Plugins
We provide a plugin for Headlamp to enhance the user experience when using Capsule. The plugin is available in the Headlamp Plugin Repository.
Offers great UX for both Capsule Admins and Tenant Owners. The plugin provides a dedicated view for Capsule Tenants, allowing users to easily manage their resources and view their usage. It also provides a dedicated view for Capsule Admins, allowing them to easily manage Tenants and view their usage.
1.8 - Kyverno
Capsule integration with Kyverno
Kyverno is a policy engine designed for Kubernetes. It provides the ability to validate, mutate, and generate Kubernetes resources using admission control. Kyverno policies are managed as Kubernetes resources and can be applied to a cluster using kubectl. Capsule integrates with Kyverno to provide a set of policies that can be used to improve the security and governance of the Kubernetes cluster.
Permissions
Some policies are attempting to query Capsule specific information, such as the tenant name based on the namespace. Therefore, we need to ensure that Kyverno has the necessary permissions to read Capsule resources. This can be achieved by extending the Kyverno ClusterRole to include Capsule resources:
admissionController:
rbac:
clusterRole:
extraResources:
- apiGroups: ["capsule.clastix.io"]
resources: ["*"]
verbs: ["get", "list"]
Recommended Policies
Not all relevant settings are covered by Capsule. We recommend to use Kyverno to enforce additional policies, as their policy implementation is of a very high standard. Here are some policies you might want to consider in multi-tenant environments:
Moved to new page
References
Here are some policies for reference. We do not provide a complete list of policies, but we provide some examples to get you started. This policies are not meant to be used in production. You may adopt principles shown here to create your own policies.
To get the tenant name based on the namespace, you can use a context. With this context we resolve the tenant, based on the {{request.namespace}} for the requested resource. The context calls /api/v1/namespaces/ API with the {{request.namespace}}. The jmesPath is used to check if the tenant label is present. You could assign a default if nothing was found, in this case it’s empty string:
context:
- name: tenant_name
apiCall:
method: GET
urlPath: "/api/v1/namespaces/{{request.namespace}}"
jmesPath: "not_null(metadata.labels.\"capsule.clastix.io/tenant\" || '')"
Select namespaces with label capsule.clastix.io/tenant
When you are performing a policy on namespaced objects, you can select the objects, which are within a tenant namespace by using the namespaceSelector. In this example we select all Kustomization and HelmRelease resources which are within a tenant namespace:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: flux-policies
spec:
validationFailureAction: Enforce
rules:
# Enforcement (Mutate to Default)
- name: Defaults Kustomizations/HelmReleases
match:
any:
- resources:
kinds:
- Kustomization
- HelmRelease
operations:
- CREATE
- UPDATE
namespaceSelector:
matchExpressions:
- key: "capsule.clastix.io/tenant"
operator: Exists
mutate:
patchStrategicMerge:
spec:
+(targetNamespace): "{{ request.object.metadata.namespace }}"
+(serviceAccountName): "default"
Compare Source and Destination Tenant
With this policy we try to enforce, that helmreleases within a tenant can only use targetNamespaces, which are within the same tenant or the same namespace the resource is deployed in:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tenant-compare
spec:
validationFailureAction: Enforce
background: true
rules:
- name: Validate HelmRelease/Kustomization Target Namespace
context:
# Get tenant based on target namespace
- name: destination_tenant
apiCall:
urlPath: "/api/v1/namespaces/{{request.object.spec.targetNamespace}}"
jmesPath: "metadata.labels.\"capsule.clastix.io/tenant\""
# Get tenant based on resource namespace
- name: source_tenant
apiCall:
urlPath: "/api/v1/namespaces/{{request.object.metadata.namespace}}"
jmesPath: "metadata.labels.\"capsule.clastix.io/tenant\""
match:
any:
- resources:
kinds:
- HelmRelease
- Kustomization
operations:
- CREATE
- UPDATE
namespaceSelector:
matchExpressions:
- key: "capsule.clastix.io/tenant"
operator: Exists
preconditions:
all:
- key: "{{request.object.spec.targetNamespace}}"
operator: NotIn
values: [ "{{request.object.metadata.namespace}}" ]
validate:
message: "spec.targetNamespace must be in the same tenant ({{source_tenant}})"
deny:
conditions:
- key: "{{source_tenant}}"
operator: NotEquals
value: "{{destination_tenant}}"
Using Global Configuration
When creating a lot of policies, you might want to abstract your configuration into a global configuration. This is a good practice to avoid duplication and to have a single source of truth. Also if we introduce breaking changes (like changing the label name), we only have to change it in one place. Here is an example of a global configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: kyverno-global-config
namespace: kyverno-system
data:
# Label for public namespaces
public_identifier_label: "company.com/public"
# Value for Label for public namespaces
public_identifier_value: "yeet"
# Label which is used to select the tenant name
tenant_identifier_label: "capsule.clastix.io/tenant"
This configuration can be referenced via context in your policies. Let’s extend the above policy with the global configuration. Additionally we would like to allow the usage of public namespaces:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tenant-compare
spec:
validationFailureAction: Enforce
background: true
rules:
- name: Validate HelmRelease/Kustomization Target Namespace
context:
# Load Global Configuration
- name: global
configMap:
name: kyverno-global-config
namespace: kyverno-system
# Get All Public Namespaces based on the label and it's value from the global configuration
- name: public_namespaces
apiCall:
urlPath: "/api/v1/namespaces"
jmesPath: "items[?metadata.labels.\"{{global.data.public_identifier_label}}\" == '{{global.data.public_identifier_value}}'].metadata.name | []"
# Get Tenant information from source namespace
# Defaults to a character, which can't be a label value
- name: source_tenant
apiCall:
urlPath: "/api/v1/namespaces/{{request.object.metadata.namespace}}"
jmesPath: "metadata.labels.\"{{global.data.tenant_identifier_label}}\" | '?'"
# Get Tenant information from destination namespace
# Returns Array with Tenant Name or Empty
- name: destination_tenant
apiCall:
urlPath: "/api/v1/namespaces"
jmesPath: "items[?metadata.name == '{{request.object.spec.targetNamespace}}'].metadata.labels.\"{{global.data.tenant_identifier_label}}\""
preconditions:
all:
- key: "{{request.object.spec.targetNamespace}}"
operator: NotIn
values: [ "{{request.object.metadata.namespace}}" ]
any:
# Source is not Self-Reference
- key: "{{request.object.spec.targetNamespace}}"
operator: NotEquals
value: "{{request.object.metadata.namespace}}"
# Source not in Public Namespaces
- key: "{{request.object.spec.targetNamespace}}"
operator: NotIn
value: "{{public_namespaces}}"
# Source not in Destination
- key: "{{request.object.spec.targetNamespace}}"
operator: NotIn
value: "{{destination_tenant}}"
match:
any:
- resources:
kinds:
- HelmRelease
- Kustomization
operations:
- CREATE
- UPDATE
namespaceSelector:
matchExpressions:
- key: "capsule.clastix.io/tenant"
operator: Exists
validate:
message: "Can not use namespace {{request.object.spec.chart.spec.sourceRef.namespace}} as source reference!"
deny: {}
Extended Validation and Defaulting
Here’s extended examples for using validation and defaulting. The first policy is used to validate the tenant name. The second policy is used to default the tenant properties, you as cluster-administrator would like to enforce for each tenant.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tenant-core
spec:
validationFailureAction: Enforce
rules:
- name: tenant-name
match:
all:
- resources:
kinds:
- "capsule.clastix.io/v1beta2/Tenant"
operations:
- CREATE
- UPDATE
validate:
message: "Using this tenant name is not allowed."
deny:
conditions:
- key: "{{ request.object.metadata.name }}"
operator: In
value: ["default", "cluster-system" ]
- name: tenant-properties
match:
any:
- resources:
kinds:
- "capsule.clastix.io/v1beta2/Tenant"
operations:
- CREATE
- UPDATE
mutate:
patchesJson6902: |-
- op: add
path: "/spec/namespaceOptions/forbiddenLabels/deniedRegex"
value: ".*company.ch"
- op: add
path: "/spec/priorityClasses/matchLabels"
value:
consumer: "customer"
- op: add
path: "/spec/serviceOptions/allowedServices/nodePort"
value: false
- op: add
path: "/spec/ingressOptions/allowedClasses/matchLabels"
value:
consumer: "customer"
- op: add
path: "/spec/storageClasses/matchLabels"
value:
consumer: "customer"
- op: add
path: "/spec/nodeSelector"
value:
nodepool: "workers"
Adding Default Owners/Permissions to Tenant
Since the Owners Spec is a list, it’s a bit more trickier to add a default owner without causing recursions. You must make sure, to validate if the value you are setting is already present. Otherwise you will create a loop. Here is an example of a policy, which adds the cluster:admin as owner to a tenant:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: tenant-policy
spec:
validationFailureAction: Enforce
background: true
rules:
# With this policy for each tenant cluster:admin is added as owner.
# Only Append these on CREATE, otherwise they will be added per reconciliation and create a loop.
- name: tenant-owner
preconditions:
all:
- key: "cluster:admin"
operator: NotIn
value: "{{ request.object.spec.owners[?kind == 'Group'].name }}"
match:
all:
- resources:
kinds:
- "capsule.clastix.io/v1beta2/Tenant"
operations:
- CREATE
- UPDATE
mutate:
patchesJson6902: |-
- op: add
path: "/spec/owners/-"
value:
name: "cluster:admin"
kind: "Group"
# With this policy for each tenant a default ProxySettings are added.
# Completely overwrites the ProxySettings, if they are already present.
- name: tenant-proxy-settings
match:
any:
- resources:
kinds:
- "capsule.clastix.io/v1beta2/Tenant"
operations:
- CREATE
- UPDATE
mutate:
foreach:
- list: "request.object.spec.owners"
patchesJson6902: |-
- path: /spec/owners/{{elementIndex}}/proxySettings
op: add
value:
- kind: IngressClasses
operations:
- List
- kind: StorageClasses
operations:
- List
- kind: PriorityClasses
operations:
- List
- kind: Nodes
operations:
- List
1.9 - Lens
Capsule extension for Lens
With Capsule extension for Lens, a cluster administrator can easily manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.
Features
Capsule extension for Lens provides these capabilities:
- List all tenants
- See tenant details and change through the embedded Lens editor
- Check Resources Quota and Budget at both the tenant and namespace level
Please, see the README for details about the installation of the Capsule Lens Extension.
1.10 - Monitoring
Capsule integration with Monitoring Solutions
While we can not provide a full list of all the monitoring solutions available, we can provide some guidance on how to integrate Capsule with some of the most popular ones. Also this is dependent on how you have set up your monitoring solution. We will just explore the options available to you.
Logging
Loki
Promtail
config:
clients:
- url: "https://loki.company.com/loki/api/v1/push"
# Maximum wait period before sending batch
batchwait: 1s
# Maximum batch size to accrue before sending, unit is byte
batchsize: 102400
# Maximum time to wait for server to respond to a request
timeout: 10s
backoff_config:
# Initial backoff time between retries
min_period: 100ms
# Maximum backoff time between retries
max_period: 5s
# Maximum number of retries when sending batches, 0 means infinite retries
max_retries: 20
tenant_id: "tenant"
external_labels:
cluster: "${cluster_name}"
serverPort: 3101
positions:
filename: /run/promtail/positions.yaml
target_config:
# Period to resync directories being watched and files being tailed
sync_period: 10s
snippets:
pipelineStages:
- docker: {}
# Drop health logs
- drop:
expression: "(.*/health-check.*)|(.*/health.*)|(.*kube-probe.*)"
- static_labels:
cluster: ${cluster}
- tenant:
source: tenant
# This wont work if pods on the cluster are not labeled with tenant
extraRelabelConfigs:
- action: replace
source_labels:
- __meta_kubernetes_pod_label_capsule_clastix_io_tenant
target_label: tenant
...
As mentioned, the above configuration will not work if the pods on the cluster are not labeled with tenant. You can use the following Kyverno policy to ensure that all pods are labeled with tenant. If the pod does not belong to any tenant, it will be labeled with management (assuming you have a central management tenant)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: capsule-pod-labels
spec:
background: false
rules:
- name: add-pod-label
context:
- name: tenant
apiCall:
method: GET
urlPath: "/api/v1/namespaces/{{request.namespace}}"
jmesPath: "not_null(metadata.labels.\"capsule.clastix.io/tenant\" || 'management')"
match:
all:
- resources:
kinds:
- Pod
operations:
- CREATE
- UPDATE
mutate:
patchStrategicMerge:
metadata:
labels:
+(capsule.clastix.io/tenant): "{{ tenant_name }}"
Grafana
1.11 - OpenCost
OpenCost Integration for Tenants
This guide explains how to integrate OpenCost with Capsule to provide cost visibility and chargeback/showback per tenant.
You can group workloads into tenants by annotating namespaces (for example, opencost.projectcapsule.dev/tenant: {{ tenant.name }}).
OpenCost can use this annotation to aggregate costs, enabling accurate cost allocation across clusters, nodes, namespaces, controller kinds, controllers, services, pods, and containers for each tenant.
Prerequisites
Installation
Capsule
- Create a tenant with spec.namespaceOptions.additionalMetadataList:
kubectl create -f - << EOF
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
namespaceOptions:
additionalMetadataList:
- annotations:
opencost.projectcapsule.dev/tenant: "{{ tenant.name }}"
owners:
- name: alice
kind: User
EOF
OpenCost
- Create a basic OpenCost values file. Set emitNamespaceAnnotations: true because aggregation is based on the Capsule annotation.
opencost:
prometheus:
internal:
namespaceName: prometheus-system
serviceName: prometheus-server
port: 80
dataRetention:
dailyResolutionDays: 30 # default: 15
exporter:
defaultClusterId: kind-opencost-capsule
replicas: 1
resources:
requests:
cpu: "10m"
memory: "55Mi"
limits:
memory: "1Gi"
persistence:
enabled: false
metrics:
kubeStateMetrics:
emitNamespaceAnnotations: true
emitPodAnnotations: true
emitKsmV1Metrics: false
emitKsmV1MetricsOnly: false
serviceMonitor:
enabled: true
additionalLabels:
release: prometheus
- Install OpenCost with the values above:
helm install opencost opencost-charts/opencost --namespace opencost --create-namespace -f values.yaml
Fetch data from OpenCost
Note!
Aggregation is only possible via the API. There is no option to aggregate using the UI.- Port-forward:
kubectl -n opencost port-forward deployment/opencost 9003:9003 9090:9090
- Query the API:
- Aggregate by namespace:
curl -G http://localhost:9003/allocation \
-d window=1h \
-d aggregate=namespace,annotation:opencost_projectcapsule_dev_tenant \
-d resolution=1h
- Aggregate by pod:
curl -G http://localhost:9003/allocation \
-d window=1h \
-d aggregate=pod,annotation:opencost_projectcapsule_dev_tenant \
-d resolution=1h
- Aggregate by deployment:
curl -G http://localhost:9003/allocation \
-d window=1h \
-d aggregate=deployment,annotation:opencost_projectcapsule_dev_tenant \
-d resolution=1h
1.12 - OpenSearch
Provision OpenSearch tenants, roles, index policies, and writer credentials with Capsule.
OpenSearch can provide a shared logging service for Capsule Tenants. The OpenSearch Kubernetes Operator manages the cluster and its security resources, while Capsule GlobalTenantResources generate the configuration for each participating Tenant. External Secrets Operator supplies passwords from an external secret backend.
This example first installs a shared cluster, then provisions the following resources for a Capsule Tenant named solar:
| Resource | Purpose |
|---|
OpensearchTenant | A Dashboards workspace named solar. |
OpensearchRole | Separate permissions for owners, members, viewers, and writers. |
OpensearchUserRoleBinding | Connect the three human access groups and the static writer user to their roles. |
OpenSearchISMPolicy | Delete the Tenant’s log indices after 30 days. |
ExternalSecret and OpensearchUser | Maintain the tenant-solar-writer account using a password stored outside Kubernetes. |
A Capsule Tenant groups Kubernetes namespaces. An OpenSearch tenant holds Dashboards saved objects, such as visualizations and index patterns. Index access is controlled separately by OpenSearch roles. This guide configures both boundaries.
flowchart LR
Tenant[Capsule Tenant] --> GTR[GlobalTenantResources]
GTR --> Security[OpenSearch tenant, roles, and bindings]
GTR --> ISM[ISM policy]
GTR --> User[Writer user]
GTR --> ES[ExternalSecret]
Backend[External secret backend] --> ES
ES --> Secret[Kubernetes Secret]
Secret --> User
Security --> Operator[OpenSearch Operator]
ISM --> Operator
User --> Operator
Operator --> Cluster[Shared OpenSearch cluster]
All generated objects live in opensearch-system, a platform-owned namespace outside any Capsule Tenant. Tenant owners must not have Kubernetes write access there: the OpenSearch custom resources can grant access to the shared cluster. In the GlobalTenantResource, set the scope to Tenant. That renders each set of resources once per enabled Capsule Tenant, regardless of how many namespaces it owns.
Prerequisites
Run the example as a cluster administrator. You need:
- Capsule with support for
GlobalTenantResource.spec.scope: Tenant, generators, and Tenant.spec.data; see the installation guide and templating documentation. - External Secrets Operator serving
external-secrets.io/v1, with a working ClusterSecretStore named platform-opensearch connected to your secret backend. - A default StorageClass and capacity for three OpenSearch nodes, each requesting 2 GiB of memory and a 20 GiB volume, plus Dashboards.
Helm, kubectl, curl, and jq locally.
The example pins OpenSearch Operator chart 3.0.2 and OpenSearch/Dashboards 3.3.0. It uses the opensearch.org/v1 APIs. Older examples using opensearch.opster.io/v1 need the operator’s API migration guidance.
Set up the OpenSearch cluster
Install the operator
helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/
helm repo update opensearch-operator
helm upgrade --install opensearch-operator opensearch-operator/opensearch-operator \
--version 3.0.2 \
--namespace opensearch-operator-system --create-namespace --wait
kubectl create namespace opensearch-system
The chart’s default watch scope includes opensearch-system. If you restrict the operator’s watched namespaces, include this namespace.
Create the shared cluster
Save this as cluster.yaml:
apiVersion: opensearch.org/v1
kind: OpenSearchCluster
metadata:
name: shared-search
namespace: opensearch-system
spec:
general:
serviceName: shared-search
version: "3.3.0"
httpPort: 9200
setVMMaxMapCount: true
security:
tls:
transport:
generate: true
perNode: true
http:
generate: true
dashboards:
enable: true
version: "3.3.0"
replicas: 1
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
memory: 1Gi
nodePools:
- component: nodes
replicas: 3
diskSize: 20Gi
jvm: "-Xms1g -Xmx1g"
roles:
- cluster_manager
- data
- ingest
resources:
requests:
cpu: 500m
memory: 2Gi
limits:
memory: 2Gi
kubectl apply -f cluster.yaml
kubectl -n opensearch-system get pods -w
Wait for the three OpenSearch nodes and Dashboards to become ready. The operator generates TLS certificates and random credentials in shared-search-admin-password and shared-search-dashboards-password. Its init container sets vm.max_map_count; if your cluster disallows that privileged operation, configure the node setting separately and set setVMMaxMapCount: false. See the pinned operator guide.
This cluster uses the operator’s bundled security configuration to demonstrate provisioning. Before making it available to tenants, review its default users, role mappings, and authentication settings. Avoid managing the same users, roles, or tenants through both a security configuration Secret and operator CRs: applying the configuration can overwrite CR-managed settings. The operator guide explains this interaction under user and role management.
Prepare tenant provisioning
Restrict the secret store
Use a dedicated platform-opensearch store. Configure its existing ClusterSecretStore with the following spec.conditions, keeping its provider and authentication configuration:
spec:
conditions:
- namespaces:
- opensearch-system
This prevents Tenant namespaces from referencing a store that can read every writer password. Conditions are alternatives, so remove broader conditions from this dedicated store. Provider configuration depends on your backend; see ClusterSecretStore.
In that backend, create a record at opensearch/tenants/solar/writer containing a password property with a strong, unique password. Paths in remoteRef.key are relative to the store’s provider configuration, such as a Vault mount. Create a separate record for every Tenant you enable. External Secrets synchronizes an existing password here; it does not generate the backend record.
Give Capsule permission to generate the resources
Use a dedicated replication ServiceAccount. Save this as provisioner.yaml and apply it:
apiVersion: v1
kind: ServiceAccount
metadata:
name: capsule-opensearch
namespace: opensearch-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: capsule-opensearch
namespace: opensearch-system
rules:
- apiGroups: ["opensearch.org"]
resources:
- opensearchtenants
- opensearchroles
- opensearchuserrolebindings
- opensearchismpolicies
- opensearchusers
verbs: ["get", "list", "create", "patch", "delete"]
- apiGroups: ["external-secrets.io"]
resources: ["externalsecrets"]
verbs: ["get", "list", "create", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: capsule-opensearch
namespace: opensearch-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: capsule-opensearch
subjects:
- kind: ServiceAccount
name: capsule-opensearch
namespace: opensearch-system
kubectl apply -f provisioner.yaml
The ServiceAccount manages only the listed custom resources in opensearch-system. External Secrets writes the password Secrets, and the OpenSearch Operator reads them using its own permissions.
Enable OpenSearch for a Capsule Tenant
Save this as tenant.yaml. For an existing Tenant, add spec.data.opensearch to its current manifest, preserving its owners and other settings.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- kind: Group
name: tenant-solar-owners
data:
opensearch:
enabled: true
retention: 30d
kubectl apply -f tenant.yaml
spec.data.opensearch.enabled is a boolean switch. All three generators below default it to false when it is absent, including when the entire opensearch or data section is missing. Set it to true to provision access, retention, and writer credentials together. retention is required only for enabled Tenants.
Role names and backend group names are derived from metadata.name: tenant-solar-owners, tenant-solar-members, and tenant-solar-viewers. Configure your identity provider to supply these group names to OpenSearch. Capsule’s Kubernetes owner assignment does not authenticate a group to OpenSearch or assign its OpenSearch role. The static writer account uses OpenSearch’s internal user database and can be tested independently.
The platform controls the enable switch and retention configuration. Keep Tenant names stable: they determine resource names, identity groups, credential paths, and index patterns. No role names are read from Tenant data.
Generate the OpenSearch tenant, roles, and bindings
Save this as opensearch-access.yaml. The four roles provide the following access:
| Role | Index permissions | Dashboards workspace |
|---|
tenant-solar-writers | Read, ingest, update, delete, and manage ingestion-related operations on live log indices. | No access. |
tenant-solar-owners | Read and manage live log indices and restored snapshot indices, including index deletion. | Read and write saved objects. |
tenant-solar-members | Read and monitor live log indices and restored snapshot indices. | Read and write saved objects. |
tenant-solar-viewers | Read and monitor live log indices and restored snapshot indices. | Read saved objects. |
The writer role supports shippers that also read, refresh, manage aliases, and delete data. It is a broader service role than an ingestion-only account. Owners manage indices; manage does not itself grant document ingestion permissions.
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: opensearch-access
spec:
scope: Tenant
resyncPeriod: 60s
serviceAccount:
name: capsule-opensearch
namespace: opensearch-system
resources:
- additionalMetadata:
labels:
observability.example.com/tenant: "{{tenant.name}}"
app.kubernetes.io/name: "{{tenant.name}}"
app.kubernetes.io/part-of: tenant
app.kubernetes.io/component: logging
generators:
- missingKey: error
template: |
{{- $spec := index $.tenant "spec" | default dict }}
{{- $data := index $spec "data" | default dict }}
{{- $opensearch := index $data "opensearch" | default dict }}
{{- if (index $opensearch "enabled") }}
apiVersion: opensearch.org/v1
kind: OpensearchTenant
metadata:
name: {{ $.tenant.metadata.name | quote }}
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
description: "Dashboards workspace for Capsule Tenant {{ $.tenant.metadata.name }}"
---
apiVersion: opensearch.org/v1
kind: OpensearchRole
metadata:
name: tenant-{{ $.tenant.metadata.name }}-writers
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
clusterPermissions:
- cluster_composite_ops
- cluster_monitor
indexPermissions:
- indexPatterns:
- "tenant-{{ $.tenant.metadata.name }}_*"
allowedActions:
- read
- index
- create_index
- "indices:admin/get"
- "indices:admin/refresh*"
- "indices:admin/delete"
- "indices:admin/auto_create"
- "indices:admin/aliases*"
- "indices:admin/exists"
- "indices:admin/mappings/get"
- "indices:data/write/bulk*"
- "indices:data/write/delete*"
- "indices:data/read/scroll"
---
apiVersion: opensearch.org/v1
kind: OpensearchRole
metadata:
name: tenant-{{ $.tenant.metadata.name }}-owners
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
clusterPermissions:
- cluster_composite_ops
- cluster_monitor
indexPermissions:
- indexPatterns:
- "tenant-{{ $.tenant.metadata.name }}_*"
- "remote_snapshot_tenant-{{ $.tenant.metadata.name }}_*"
allowedActions:
- read
- manage
tenantPermissions:
- tenantPatterns:
- {{ $.tenant.metadata.name | quote }}
allowedActions:
- kibana_all_read
- kibana_all_write
---
apiVersion: opensearch.org/v1
kind: OpensearchRole
metadata:
name: tenant-{{ $.tenant.metadata.name }}-members
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
clusterPermissions:
- cluster_composite_ops
indexPermissions:
- indexPatterns:
- "tenant-{{ $.tenant.metadata.name }}_*"
- "remote_snapshot_tenant-{{ $.tenant.metadata.name }}_*"
allowedActions:
- read
- indices_monitor
tenantPermissions:
- tenantPatterns:
- {{ $.tenant.metadata.name | quote }}
allowedActions:
- kibana_all_read
- kibana_all_write
---
apiVersion: opensearch.org/v1
kind: OpensearchRole
metadata:
name: tenant-{{ $.tenant.metadata.name }}-viewers
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
clusterPermissions:
- cluster_composite_ops_ro
indexPermissions:
- indexPatterns:
- "tenant-{{ $.tenant.metadata.name }}_*"
- "remote_snapshot_tenant-{{ $.tenant.metadata.name }}_*"
allowedActions:
- read
- indices_monitor
tenantPermissions:
- tenantPatterns:
- {{ $.tenant.metadata.name | quote }}
allowedActions:
- kibana_all_read
{{- range $role := list "owners" "members" "viewers" }}
---
apiVersion: opensearch.org/v1
kind: OpensearchUserRoleBinding
metadata:
name: tenant-{{ $.tenant.metadata.name }}-{{ $role }}
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
backendRoles:
- tenant-{{ $.tenant.metadata.name }}-{{ $role }}
roles:
- tenant-{{ $.tenant.metadata.name }}-{{ $role }}
- kibana_user
{{- if eq $role "viewers" }}
- kibana_read_only
{{- end }}
{{- end }}
---
apiVersion: opensearch.org/v1
kind: OpensearchUserRoleBinding
metadata:
name: tenant-{{ $.tenant.metadata.name }}-writers
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
users:
- tenant-{{ $.tenant.metadata.name }}-writer
roles:
- tenant-{{ $.tenant.metadata.name }}-writers
{{- end }}
The underscore in tenant-solar_* separates the Tenant name from the rest of the index name. Capsule Tenant names cannot contain underscores, so solar and solar-prod receive distinct patterns. A pattern such as tenant-solar-* would also match tenant-solar-prod-*. Use the underscore convention consistently in shippers, policies, and snapshot restore names. The remote_snapshot_tenant-solar_* pattern permits access to restored indices following that convention; this guide does not create snapshots or restore them.
The human bindings include the built-in kibana_user role for Dashboards access. The custom roles use tenantPermissions for the named workspace, without adding broad .dashboard_* or .kibana* patterns. Viewers also receive kibana_read_only for the default Dashboards read-only UI mode. Their index and tenant permissions enforce read-only access independently of that UI setting. Permissions from multiple roles accumulate, so a viewer who also belongs to the owners group still has owner permissions through the API. See users and roles.
indices_monitor belongs under indexPermissions, where it is scoped to the Tenant’s indices. cluster_monitor grants cluster-wide monitoring to owners and writers. The writer’s wildcard actions include refresh and delete-by-query operations. See the action group definitions.
If your platform defines a custom action group such as cluster-reports-access, add it to the owners’ and members’ clusterPermissions after provisioning that group. It is not a built-in action group, so this example does not depend on it. Review report access separately from index and Dashboards tenant access.
kubectl apply -f opensearch-access.yaml
Generate ISM policies
Index State Management manages the lifetime of indices. This example uses daily indices such as tenant-solar_2026.09.22, then deletes each index when its age reaches the Tenant’s retention period. Index age is measured from index creation, not from timestamps inside documents. Restored indices using the remote_snapshot_tenant- prefix do not match this retention policy.
Save this as opensearch-retention.yaml:
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: opensearch-retention
spec:
scope: Tenant
resyncPeriod: 60s
serviceAccount:
name: capsule-opensearch
namespace: opensearch-system
resources:
- additionalMetadata:
labels:
observability.example.com/tenant: "{{tenant.name}}"
app.kubernetes.io/name: "{{tenant.name}}"
app.kubernetes.io/part-of: tenant
app.kubernetes.io/component: logging
generators:
- missingKey: error
template: |
{{- $spec := index $.tenant "spec" | default dict }}
{{- $data := index $spec "data" | default dict }}
{{- $opensearch := index $data "opensearch" | default dict }}
{{- if (index $opensearch "enabled") }}
apiVersion: opensearch.org/v1
kind: OpenSearchISMPolicy
metadata:
name: tenant-{{ $.tenant.metadata.name }}-logs
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
description: "Log retention for Capsule Tenant {{ $.tenant.metadata.name }}"
defaultState: hot
applyToExistingIndices: false
ismTemplate:
indexPatterns:
- "tenant-{{ $.tenant.metadata.name }}_*"
priority: 100
states:
- name: hot
actions: []
transitions:
- stateName: delete
conditions:
minIndexAge: {{ $.tenant.spec.data.opensearch.retention | quote }}
- name: delete
actions:
- delete: {}
{{- end }}
kubectl apply -f opensearch-retention.yaml
ismTemplate attaches the policy to newly created matching indices. applyToExistingIndices: false leaves older indices untouched. Enable it only when you intend to apply retention to existing data. Changing a policy does not necessarily change every already-managed index immediately; inspect ISM state and use the ISM APIs when migrating existing indices. This policy does not perform rollover: the shipper must select a new daily index.
Generate static writer users with External Secrets
Save this as opensearch-writers.yaml. For each enabled Tenant, Capsule creates an ExternalSecret and an OpensearchUser. External Secrets reads the password into a Secret, and the OpenSearch Operator uses passwordFrom to configure the internal user. The access generator binds the singular tenant-solar-writer user to the plural tenant-solar-writers role.
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: opensearch-writers
spec:
scope: Tenant
resyncPeriod: 60s
serviceAccount:
name: capsule-opensearch
namespace: opensearch-system
resources:
- additionalMetadata:
labels:
observability.example.com/tenant: "{{tenant.name}}"
app.kubernetes.io/name: "{{tenant.name}}"
app.kubernetes.io/part-of: tenant
app.kubernetes.io/component: logging
generators:
- missingKey: error
template: |
{{- $spec := index $.tenant "spec" | default dict }}
{{- $data := index $spec "data" | default dict }}
{{- $opensearch := index $data "opensearch" | default dict }}
{{- if (index $opensearch "enabled") }}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: tenant-{{ $.tenant.metadata.name }}-writer
namespace: opensearch-system
spec:
refreshPolicy: Periodic
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: platform-opensearch
target:
name: tenant-{{ $.tenant.metadata.name }}-writer
creationPolicy: Owner
template:
engineVersion: v2
mergePolicy: Merge
metadata:
annotations:
opensearchuser/name: tenant-{{ $.tenant.metadata.name }}-writer
opensearchuser/namespace: opensearch-system
data:
username: tenant-{{ $.tenant.metadata.name }}-writer
endpoint: https://shared-search.opensearch-system.svc:9200
data:
- secretKey: password
remoteRef:
key: opensearch/tenants/{{ $.tenant.metadata.name }}/writer
property: password
---
apiVersion: opensearch.org/v1
kind: OpensearchUser
metadata:
name: tenant-{{ $.tenant.metadata.name }}-writer
namespace: opensearch-system
spec:
opensearchCluster:
name: shared-search
passwordFrom:
name: tenant-{{ $.tenant.metadata.name }}-writer
key: password
{{- end }}
kubectl apply -f opensearch-writers.yaml
ESO’s mergePolicy: Merge preserves the fetched password alongside the generated username and endpoint. No password passes through the Capsule template or needs to be committed to Git.
The Secret annotations identify the user to reconcile when its password changes. Set both explicitly because this Secret has multiple data keys; otherwise the operator’s multi-user Secret handling can interpret those keys as usernames. This follows the pinned operator’s Secret watch implementation.
The OpenSearch resources and their password Secrets use the cluster’s namespace. Their opensearchCluster and passwordFrom references are local references. The controllers reconcile asynchronously, so a user or binding may initially be pending while its dependencies are created.
Platform-managed log shippers in opensearch-system can mount the corresponding writer Secret. Configure each output with that Tenant’s credentials and index prefix, and trust the cluster’s public CA certificate. Route logs using trusted Kubernetes namespace ownership, rather than a tenant-supplied field in a log message. If shippers run in Tenant namespaces, distribute only the matching writer Secret through a separately scoped GlobalTenantResource replication; keep the platform secret store restricted.
Verify the integration
First check Capsule’s generation and each downstream controller:
kubectl wait --for=condition=Ready --timeout=120s \
globaltenantresource/opensearch-access \
globaltenantresource/opensearch-retention \
globaltenantresource/opensearch-writers
kubectl -n opensearch-system wait --for=condition=Ready --timeout=120s \
externalsecret/tenant-solar-writer
kubectl -n opensearch-system get \
opensearchtenants.opensearch.org,opensearchroles.opensearch.org,opensearchuserrolebindings.opensearch.org,opensearchismpolicies.opensearch.org,opensearchusers.opensearch.org
Capsule’s Ready condition means the Kubernetes objects were reconciled. Wait for the OpenSearch resources to report status.state: CREATED before testing access. If they do not, inspect their status.reason and controller logs. Existing unmanaged objects with the same names can prevent the operator from taking ownership; use fresh names for this example.
In a separate terminal, expose the REST service locally:
kubectl -n opensearch-system port-forward svc/shared-search 9200:9200
Fetch only the public CA certificate, then use the service’s certificate hostname with the forwarded connection:
kubectl -n opensearch-system get secret shared-search-http-cert -o json \
| jq -r '.data["ca.crt"] | @base64d' > opensearch-ca.crt
OS_URL=https://shared-search.opensearch-system.svc:9200
OS_CONNECT=shared-search.opensearch-system.svc:9200:localhost:9200
The following commands prompt for the writer password from your external backend; they do not place it in shell history. Writing its own Tenant’s logs should return HTTP 201, writing another Tenant’s logs should return HTTP 403, and searching its own logs should return HTTP 200:
# Allowed: ingest into this Tenant's index.
curl --cacert opensearch-ca.crt --connect-to "$OS_CONNECT" \
--user tenant-solar-writer -w '\nHTTP %{http_code}\n' \
-H 'Content-Type: application/json' \
-X POST "$OS_URL/tenant-solar_2026.09.22/_doc" \
-d '{"message":"hello from solar"}'
# Denied: ingest into another Tenant's index.
curl --cacert opensearch-ca.crt --connect-to "$OS_CONNECT" \
--user tenant-solar-writer -w '\nHTTP %{http_code}\n' \
-H 'Content-Type: application/json' \
-X POST "$OS_URL/tenant-wind_2026.09.22/_doc" \
-d '{"message":"must be rejected"}'
# Allowed: this writer role includes read access to its Tenant's logs.
curl --cacert opensearch-ca.crt --connect-to "$OS_CONNECT" \
--user tenant-solar-writer -w '\nHTTP %{http_code}\n' \
"$OS_URL/tenant-solar_2026.09.22/_search"
As a platform administrator, inspect the index’s ISM policy. Retrieve the bootstrap admin password from shared-search-admin-password and enter it at the prompt:
curl --cacert opensearch-ca.crt --connect-to "$OS_CONNECT" \
--user admin \
"$OS_URL/_plugins/_ism/explain/tenant-solar_2026.09.22?show_policy=true"
Allow time for ISM’s background processing. The response should identify tenant-solar-logs as the policy. For interactive access, forward svc/shared-search-dashboards on port 5601, authenticate through your configured identity provider, and select solar. As an owner or member, create an index pattern for tenant-solar_* and save a visualization. As a viewer, verify that you can read those saved objects but cannot edit them. Use separate identities for each access level, since OpenSearch combines their role permissions. Verify that none of these identities can access another Tenant’s logs or workspace.
Rotation and tenant lifecycle
To rotate a writer password, update its existing backend record. ESO refreshes the Kubernetes Secret, and the OpenSearch Operator reconciles the internal user. You can request an immediate ESO refresh with:
kubectl -n opensearch-system annotate externalsecret tenant-solar-writer \
force-sync="$(date +%s)" --overwrite
Reload or restart shippers that do not watch credential changes. Secret updates, OpenSearch user updates, and client reloads are separate operations, so a single static account can experience a short authentication interruption during rotation.
To onboard another Tenant, create its backend password record, configure the three derived identity groups, and set spec.data.opensearch.enabled: true with a retention period. The three GlobalTenantResources provision its resources automatically.
To disable this integration for a Tenant, set the switch to false:
kubectl patch tenant solar --type=merge \
-p '{"spec":{"data":{"opensearch":{"enabled":false}}}}'
All three generators then produce no objects for that Tenant, allowing Capsule to prune previously generated resources according to its object management rules. Removing the switch has the same effect. This withdraws the desired access resources and retention policy; it is not a pause that preserves them. Keep the OpenSearch Operator and shared cluster available while security-resource finalizers run. Deleting access resources is separate from deleting stored log indices, Dashboards data, and backend password records; include those in your platform’s offboarding process and verify that access has been revoked.
1.13 - Openshift
1.14 - Rancher
1.15 - Tekton
Capsule integration with Tekton
With Capsule extension for Lens, a cluster administrator can easily manage from a single pane of glass all resources of a Kubernetes cluster, including all the Tenants created through the Capsule Operator.
Prerequisites
Tekton must be already installed on your cluster, if that’s not the case consult the documentation here:
Cluster Scoped Permissions
Tekton Dashboard
Now for the enduser experience we are going to deploy the tekton dashboard. When using oauth2-proxy we can deploy one single dashboard, which can be used for all tenants. Refer to the following guide to setup the dashboard with the oauth2-proxy:
Once that is done, we need to make small adjustments to the tekton-dashboard service account.
kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
patches:
# Adjust the service for the capsule-proxy according to your installation
# The used values are compatbile with the default installation values
- target:
version: v1
kind: Deployment
name: tekton-dashboard
patch: |-
- op: add
path: /spec/template/spec/containers/0/env/-
value:
name: KUBERNETES_SERVICE_HOST
value: "capsule-proxy.capsule-system.svc"
- op: add
path: /spec/template/spec/containers/0/env/-
value:
name: KUBERNETES_SERVICE_PORT
value: "9001"
# Adjust the CA certificate for the capsule-proxy according to your installation
- target:
version: v1
kind: Deployment
name: tekton-dashboard
patch: |-
- op: add
path: /spec/template/spec/containers/0/volumeMounts
value: []
- op: add
path: /spec/template/spec/containers/0/volumeMounts/-
value:
mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
name: token-ca
- op: add
path: /spec/template/spec/volumes
value: []
- op: add
path: /spec/template/spec/volumes/-
value:
name: token-ca
projected:
sources:
- serviceAccountToken:
expirationSeconds: 86400
path: token
- secret:
name: capsule-proxy
items:
- key: ca
path: ca.crt
This patch assumes there’s a secret called capsule-proxy with the CA certificate for the Capsule Proxy URL.
Apply the given kustomization:
extraEnv:
- name: KUBERNETES_SERVICE_HOST
value: ‘${CAPSULE_PROXY_URL}’
- name: KUBERNETES_SERVICE_PORT
value: ‘${CAPSULE_PROXY_PORT}’
Tekton Operator
When using the Tekton Operator, you need to add the following to the TektonConfig:
apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
name: config
spec:
dashboard:
readonly: false
options:
disabled: false
deployments:
tekton-dashboard:
spec:
template:
spec:
volumes:
- name: token-ca
projected:
sources:
- serviceAccountToken:
expirationSeconds: 86400
path: token
- secret:
name: capsule-proxy
items:
- key: ca
path: ca.crt
containers:
- name: tekton-dashboard
volumeMounts:
- mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
name: token-ca
env:
- name: KUBERNETES_SERVICE_HOST
value: "capsule-proxy.capsule-system.svc"
- name: KUBERNETES_SERVICE_PORT
value: "9001"
See for reference the options spec
1.16 - Teleport
Capsule Proxy integration with Teleport
Teleport is an open-source tool that provides zero trust access to servers and cloud applications using SSH, Kubernetes, Database, Remote Desktop Protocol and HTTPS. It can eliminate the need for VPNs by providing a single gateway to access computing infrastructure via SSH, Kubernetes clusters, and cloud applications via a built-in proxy.
If you want to pass requests from teleport users through the capsule-proxy for users to be able to do things like listing namespaces scoped to their own tenants, this integration is for you.
Prerequisites
- Capsule
- Capsule Proxy
- Teleport Cluster
- teleport-kube-agent
Integration
It’s recommended to install teleport-kube-agent in the capsule-system namespace. Otherwise you need to somehow replicate the internal ca secret to the namespace, where teleport-kube-agent is deployed to. For this case Cert-Manager Trust-Bundles might be useful.
Add the following values to the teleport-kube-agent helm chart and you’re already done:
extraEnv:
- name: KUBERNETES_SERVICE_HOST
value: capsule-proxy.capsule-system.svc
- name: KUBERNETES_SERVICE_PORT
value: "9001"
extraVolumes:
- name: kube-api-access-capsule
projected:
sources:
- serviceAccountToken:
path: token
- secret:
items:
- key: ca
path: ca.crt
name: capsule-proxy
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
extraVolumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-capsule
readOnly: true
Note: The secret capsule-proxy refers to the secret which is being used by the capsule-proxy instance directly, not the self-signed-ca secret.
Local Demo
If you want to test this integration locally, follow these steps.
References
The following tools have to be installed on your machine:
- docker
- kind
- kubectl
- helm
- mkcert
Docker Network
Create docker network teleport:
docker network create teleport
Self-signed certificates
Create certificates for teleport.demo:
mkdir teleport-tls
cd teleport-tls
mkcert teleport.demo "*.teleport.demo"
cp "$(mkcert -CAROOT)/rootCA.pem" .
Teleport installation
Run Ubuntu docker image in the teleport network using teleport.demo alias on port 443:
docker run -it -v .:/etc/teleport-tls --name teleport --network teleport --network-alias teleport.demo -p 443:443 ubuntu:22.04
Run the following commands inside docker container:
apt-get update && apt-get install -y curl
cp /etc/teleport-tls/rootCA.pem /etc/ssl/certs/mkcertCA.pem
curl https://cdn.teleport.dev/install.sh | bash -s 18.2.1
teleport configure -o file \
--cluster-name=teleport.demo \
--public-addr=teleport.demo:443 \
--cert-file=/etc/teleport-tls/teleport.demo+1.pem \
--key-file=/etc/teleport-tls/teleport.demo+1-key.pem
teleport start --config="/etc/teleport.yaml"
Open new shell
Note down IP of docker container:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' teleport
Kubernetes Cluster setup
- Create kind cluster:
kind create cluster --name capsule
CoreDNS
To allow pods to easily connect to the teleport service running in the other Docker container:
Connect to docker network: docker network connect teleport capsule-control-plane
Edit ConfigMap of coredns to set up dns resolution of teleport.demo: kubectl edit cm -n kube-system coredns
hosts {
<Paste IP from docker inspect command here> teleport.demo
fallthrough
}
Restart coredns Deployment: kubectl rollout restart deployment -n kube-system coredns
Capsule
capsule-values.yaml:
manager:
options:
capsuleUserGroups: ["tenant-wind"]
forceTenantPrefix: true
tenant.yaml:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: wind
spec:
owners:
- name: alice
kind: User
Install capsule with tenant-oil as a capsule user group via helm chart:
helm repo add projectcapsule https://projectcapsule.github.io/chartshelm upgrade --install capsule -n capsule-system --create-namespace projectcapsule/capsule --version 0.10.9 -f capsule-values.yaml- Create tenant named
wind: kubectl apply -f tenant.yaml
Capsule Proxy
Install default capsule-proxy via helm chart:
helm repo add projectcapsule https://projectcapsule.github.io/chartshelm upgrade --install capsule-proxy -n capsule-system projectcapsule/capsule-proxy --version 0.9.12
Teleport
Create teleport role for kubernetes cluster access which adds tenant-wind group to user auth token.
Create and set up user alice with kube-access teleport role:
tctl users add alice --roles=access,kube-access- Add
127.0.0.1 teleport.demo to etc/hosts of your computer - Set password and second factor for user
alice in browser
Create join token for teleport-kube-agent:
- Note down
authToken from command: tctl tokens add --type=kube --ttl=24h
Teleport Agent
teleport-agent-values.yaml:
proxyAddr: "teleport.demo:443"
kubeClusterName: "teleport.demo"
insecureSkipProxyTLSVerify: true
authToken: "<Paste authToken from tctl tokens add command here>"
labels:
capsule: "true"
extraEnv:
- name: KUBERNETES_SERVICE_HOST
value: capsule-proxy.capsule-system.svc
- name: KUBERNETES_SERVICE_PORT
value: "9001"
extraVolumes:
- name: kube-api-access-capsule
projected:
sources:
- serviceAccountToken:
path: token
- secret:
items:
- key: ca
path: ca.crt
name: capsule-proxy
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
extraVolumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-capsule
readOnly: true
- Update
authToken in teleport-agent-values.yaml from output of tctl tokens add command helm repo add teleport https://charts.releases.teleport.devhelm upgrade --install teleport-agent -n capsule-system teleport/teleport-kube-agent --version 18.2.0 -f teleport-agent-values.yaml
Test it out
tsh login --proxy=teleport.demo:443 --auth=local --user=alice teleport.demotsh kube login teleport.demokubectl get tenantkubectl get namespace (only works because teleport is connected to capsule-proxy instead of kubernetes api)kubectl create ns foo-bar (should fail, since not owner)kubectl create ns oil-bar (should succeed)
From here you could enable ProxyClusterScoped feature gate to allow listing of cluster scoped resources via ProxySettings.
Cleanup
kind delete clusters capsulerm -rf teleport-tlstsh logout --proxy=teleport.demo --user alicedocker rm teleport
1.17 - Velero
