https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
t
That looks right. where are you placing the file?
f
We're using the Rancher Federal Ansible installer which places the file in 
/var/lib/rancher/rke2/server/manifests/rke2_manifests.
The file is being honored and that's good
the question I have is: given out-of-the-box RKE2 config for plugins is: A+B+C, and I want to merge in Q, to get: A+B+C+Q
that's what I can't seem to do
I have to full replicate the out of the box config:
t
cool. that looks right. may be worth testing with placing the file there by hand to see if ansible is borking something. there may be a coredns bug..
f
---
apiVersion: helm.cattle.io/v1 kind: HelmChartConfig metadata: name: rke2-coredns namespace: kube-system spec: valuesContent: |- servers: - zones: - zone: . port: 53 # If serviceType is nodePort you can specify nodePort here # nodePort: 30053 # hostPort: 53 plugins: - name: errors # Serves a /health endpoint on :8080, required for livenessProbe - name: health configBlock: |- lameduck 5s # Serves a /ready endpoint on :8181, required for readinessProbe - name: ready - name: hosts configBlock: |- 172.253.122.113 baasdfljhsdafasdfasdfa.com fallthrough # Required to query kubernetes API for data - name: kubernetes parameters: cluster.local in-addr.arpa ip6.arpa configBlock: |- pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 # Serves a /metrics endpoint on :9153, required for serviceMonitor - name: prometheus parameters: 0.0.0.0:9153 - name: forward parameters: . /etc/resolv.conf - name: cache parameters: 30 - name: loop - name: reload - name: loadbalance
t
I’ll test in a few.
f
I dno't think its coredns becuase RKE2 (presumably) is building the final Core OS configmap
Core sorry
Core DNS
CoreDNS just mounts the ConfigMap
RKE2 builds the CM
so it looks to me like RKE2 doesn't expect (looking at the helm chart) to provide the ability like:
Copy code
server:
  plugin-to-add:
t
big picture, what are you trying to do?
f
heh heh - very simply this is a case where there's no DNS in a cluster so I need to provide a small number of host=IP to kubernetes so that workloads running in pods can reach remote hostnames.... This particular CoreDNS plugin allows you to essentially inline an 
/etc/hosts
file in the Corefile...
t
just to confirm this is for external resolution and not internal.
f
internal means....
t
internal to the cluster.
f
its possible that - for example - there could be four host: a.foo.org and b.foo.org
t
aka
Copy code
<namespace>.svc.cluster.local svc.cluster.local cluster.local
f
and a service might configure an ingress to be reachabe by other components inthe cluster
and so without DNS - this doesnt work
t
so you want the services to go out around back in through ingress?
f
so it's a) external and b) internal with wildcard-style DNS names
t
are you crossing namespaces?
f
the network is segmented so - if a pod does 
curl <https://xyz.a.foo.org>
as long as DNS provides an IP address and the IP address is on the same network then the packets don't need to leave the segment. 
<http://xyz.a.foo.org|xyz.a.foo.org>
just needs to be an IP address running the ingress controller... So absent DNS, one needs the equivalent of an 
/etc/hosts
file that looks like:
Copy code
12.12.12.12 <http://xyz.a.foo.org|xyz.a.foo.org>
12.12.12.12 <http://pqd.a.foo.org|pqd.a.foo.org>
etc...
(that's for intra-component service consumption)
t
where I was going is that if pod A needs to talk to pod B in the same namespace you use all internal DNS. If they are in different namespaces then it can go either internal or external through ingress. It really depends on how coupled the two namespaces/services are. What is the file name you used for the coredns yaml? I just spun up a fresh cluster and I see the 
rke2-coredns.yaml
in the dir already.
Copy code
[root@rke1 manifests]# cat rke2-coredns.yaml 
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChart
metadata:
  annotations:
    <http://helm.cattle.io/chart-url|helm.cattle.io/chart-url>: <https://rke2-charts.rancher.io/assets/rke2-coredns/rke2-coredns-1.24.002.tgz>
  name: rke2-coredns
  namespace: kube-system
spec:
  bootstrap: true
  chartContent:
f
we don't control what the apps do
t
makes sense.
ok, makes sense
f
rke2-coredns.yaml
is the one deposited by RKE2
the one under 
manifests
is deposited by the Rancher Federal installer
t
just confirming you are not overwritting it.
f
ls -l /var/lib/rancher/rke2/server/manifests/rke2_manifests
Copy code
rw-r----- 1 root root 1178 Aug  3 14:22 rke2-coredns-config.yaml
t
cool.
and you are using
Copy code
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    servers:
    - plugins:
      - name: hosts
        configBlock: |-
          fallthrough
as the file contents?
f
cat /var/lib/rancher/rke2/server/manifests/rke2_manifests/rke2-coredns-config.yaml
Copy code
---
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    servers:
    - zones:
      - zone: .
      port: 53
      # If serviceType is nodePort you can specify nodePort here
      # nodePort: 30053
      # hostPort: 53
      plugins:
      - name: errors
      # Serves a /health endpoint on :8080, required for livenessProbe
      - name: health
        configBlock: |-
          lameduck 5s
      # Serves a /ready endpoint on :8181, required for readinessProbe
      - name: ready
      - name: hosts
        configBlock: |-
          172.253.122.113 <http://baasdfljhsdafasdfasdfa.com|baasdfljhsdafasdfasdfa.com>
          fallthrough
      # Required to query kubernetes API for data
      - name: kubernetes
        parameters: cluster.local in-addr.arpa ip6.arpa
        configBlock: |-
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
          ttl 30
      # Serves a /metrics endpoint on :9153, required for serviceMonitor
      - name: prometheus
        parameters: 0.0.0.0:9153
      - name: forward
        parameters: . /etc/resolv.conf
      - name: cache
        parameters: 30
      - name: loop
      - name: reload
      - name: loadbalance
Unless I supply the FULL config - the RKE2 helm chart PLUS my plugin
it doesn't "work". Meaning - my HelmChartConfig fully replaces teh Core File in teh configmap
t
here are the defaults that I am seeing :
Copy code
clembookpro:clemenko rke2 ( 178.128.150.160:6443 ) $ helm get values -n kube-system rke2-coredns 
USER-SUPPLIED VALUES:
global:
  clusterCIDR: 10.42.0.0/16
  clusterCIDRv4: 10.42.0.0/16
  clusterDNS: 10.43.0.10
  clusterDomain: cluster.local
  rke2DataDir: /var/lib/rancher/rke2
  serviceCIDR: 10.43.0.0/16
f
yes I think everything else is defaulted:
(searching)
I'm looking for the helm chart that rke2 uses to install the core dns addon
not the overrides the actual chart with the default values
t
I just applied
Copy code
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    servers:
    - plugins:
      - name: hosts
        configBlock: |-
          fallthrough
to 
rke2-coredns-config.yaml
and restarted rke2 and got
Copy code
clembookpro:clemenko rke2 ( 178.128.150.160:6443 ) $ helm get values -n kube-system rke2-coredns 
USER-SUPPLIED VALUES:
global:
  clusterCIDR: 10.42.0.0/16
  clusterCIDRv4: 10.42.0.0/16
  clusterDNS: 10.43.0.10
  clusterDomain: cluster.local
  rke2DataDir: /var/lib/rancher/rke2
  serviceCIDR: 10.43.0.0/16
servers:
- plugins:
  - configBlock: fallthrough
    name: hosts
I wonder if the ansible is causing the issue.
f
I'm not that savvy on how Rancher builds and packages the "add-ons" a.k.a. CoreDNS into its binary
but most likely it is close to the upstream right?
t
it uses the upstream helm chart. if it is not in there rke2 can’t do much.
rke2 does not create the configmap. helm renders that.
f
sure but rke2 lays down the chart
kubectl -n kube-system get helmchart rke2-coredns -oyaml
that chart was put there by RKE2
that chart has a values. The values has a bunch of defaults
the plugins are all deafulted. If you take no action - the plugins in the link I sent line up exactly with the CM that has the 
Corefile
key
...
t
the helm chart is built into rke2. 
<http://helm.cattle.io/chart-url|helm.cattle.io/chart-url>: <https://rke2-charts.rancher.io/assets/rke2-coredns/rke2-coredns-1.24.002.tgz>
Yes.. we are using the coredns upstream chart. We didn’t make a custom chart.
f
Right - it looks to me like the 
HelmChartConfig
which is a Rancher construct was not intended as a merge patch but rather a full replacement patch
and that's ok but
t
it is.. it just worked for me. both for coredns and nginx.
f
Core DNS doens't know anyting about a HelmChartConfig
that 's a Rancher thing
t
yup.. rke2 reads it and updates the values when it renders the helm and deploys.
f
all I'm saying is it would be nice if I could patch rather than replace the plugins
right? because if I replace today and a later release of RKE2 changes that Corefile it's another thing I need to keep track of
and I probably won't so it risks breakage in the future if I upgrade that cluster
t
it does patch… when I added the file above and restarted the 
rke2-server
it updated the values and configmap.
f
kubectl -n kube-system describe cm rke2-coredns-rke2-coredns
for me - the only contents of the CM is the one plugin - the others were erased
t
retesting now.
f
let me retry with yours
t
ok, that did overwrite the configmap.
f
ok yeah that was my experience
its a replace not a patch
which makes sense - that's how yaml works
but really it would be a request for me to go back to teh CoreDNS chart maintainers and ask them to add a 
additional-plugins
key...
t
confirmed. file a GH bug. I can add to it. This behavior is not how ingress-nginx works. it does patch it there.
f
really>
t
ya.
f
ok ...
hmmmm
t
if only you had support. 😉
f
heh heh
this is my support 😉
t
ask you know who… 😎
f
sure - ok well I appreciate your time - I'll get on with life and let you do the same
t
no worries. FYI I am checking the behavor of nginx on this.
nginx had a patching nature to the CM.
f
that's my recollection having patched it in the past
Maybe I'll create a feature/bug GH issue...
👀 1
354 Views
Open in Slack
PreviousNext
Powered by