Subject: GitOps with Fleet - Dev/Prod environments...
# fleet
h
Subject: GitOps with Fleet - Dev/Prod environments on same clusters - Seeking advice Hi Fleet community! πŸ‘‹ We're learning Rancher Fleet and need advice on our GitOps setup. Our Setup: β€’ Multi-cluster application with
master/
(central components) and
fleet/
(edge components) folders β€’ Need dev/prod environments running on the same physical clusters (resource constraints) β€’ Want GitOps workflow: work on
develop
branch β†’ merge to
main
for production The Problem: When merging
develop
β†’
main
, we get conflicts because environment-specific configs (NodePorts, namespaces, experiment names) differ between environments: yaml
Copy code
# develop branch: nodePort: 30500 
# main branch:    nodePort: 31500
# After merge:    πŸ’₯ CONFLICT!
Solution 1: Using Fleet's base + overlays pattern:
Copy code
master/
β”œβ”€β”€ base/                    # Common resources (identical in both branches)
β”‚   β”œβ”€β”€ deployments/
β”‚   └── internal-services/
β”œβ”€β”€ overlays/
β”‚   β”œβ”€β”€ development/         # Dev-specific configs (ports 30xxx)
β”‚   └── production/          # Prod-specific configs (ports 31xxx)
└── fleet.yaml
Solution 1bis: Fleet's base = production and overlays apply changes only for development: Easier to manage and to implement
Copy code
main branch (Production):
β”œβ”€β”€ base/                    # Production configs (default/stable)
β”‚   β”œβ”€β”€ deployments/
β”‚   └── services/
└── fleet.yaml               # No overlays needed

develop branch (Development):
β”œβ”€β”€ base/                    # Same production configs  
β”œβ”€β”€ overlays/development/    # Dev-specific overrides only
β”‚   └── dev-configs.yaml    # Different ports, namespaces, etc.
└── fleet.yaml               # Uses development overlays
4 GitRepo resources total (master-dev, master-prod, fleet-dev, fleet-prod) pointing to different branches and using different overlays. Handling Fleet.yaml Merge Conflicts: Using
.gitattributes
with
merge=ours
to automatically keep production fleet.yaml during merges:
Copy code
# .gitattributes
*/fleet.yaml merge=ours
This ensures when merging
develop
β†’
main
, Git automatically keeps the main branch fleet.yaml (production config) and discards the develop branch version. Questions: 1. Is one of this solution the right approach for our use case? base=prod + overlay=dev approach a Fleet best practice? 2. Any issues with our
.gitattributes merge=ours
strategy for fleet.yaml files? 3. Any better patterns for same-cluster dev/prod separation? 4. Best practices for managing environment-specific configs in Fleet? 5. Alternative solutions we should consider? We're fairly new to Fleet, so any guidance from experienced users would be greatly appreciated! πŸ™ Thanks in advance!
w
1. this will always be controversial. I don't prefer overlays, but I'd lean toward number 1 to encourage clarity, even if the production folder doesn't have overrides. 2. I don't have any input, I don't like using trunks for non-ephemeral environments. 3. use folders instead of long-running branches (trunks) where possible. 4. consider separating configs from values, perhaps by separating repos. 5. comments in the above responses
h
Thanks @witty-honey-18052 for the detailed insights! πŸ™ We actually tried our Solution 1bis approach first (base = production + overlays for dev only), but ran into stability issues. The overlay mechanism got confused when trying to patch development resources that were already "baked into" the base - it was particularly problematic with complex resources like our MLflow Helm deployments and multi-port services. That's why we switched to explicit overlays for both environments (Solution 1) - it made the overlay behavior much more predictable, even if it means some duplication. Your folder-based approach opened my eyes to real Kubernetes file structuration and more sophisticated deployment patterns! The separation between
applications/
(templates) and
environments/
(values) is exactly the kind of enterprise-grade architecture I need to understand better. This brings up an interesting question though : in terms of k8s deployment, environments are not managed by git branches !? For our current lab setup, we'll stick with our working solution, but I'm definitely going to explore the folder-based + templating approach for our next project. The
.gitattributes merge=ours
is more of a "quick fix" than a proper architectural solution. Really appreciate you taking the time to explain these patterns - exactly the kind of guidance we needed as Fleet newcomers! πŸš€