https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
s
Have you thought of having an external database? I believe, and I'm sure someone will correct me if I'm wrong, but the requirement for an odd number of servers is only with using etcd for the database. https://docs.k3s.io/installation/ha
h
Yes @sticky-summer-13450 but what if the external database fails? thats still a single point of failure right?
s
You want to scale down to 1 master and you want HA - I don't think kubernetes does "cake and eat it" 😃 I gave you a solution to your request, moving the goalposts isn't a fair response.
h
roger. I think I figured it out. thanks for the input.
👍 1
@sticky-summer-13450 so I created 3 server nodes and 1 agent node in k3s cluster. external etcd DB was replicated onto each of these nodes. but this system still requires at all times that at least 2 server nodes need to be running for the cluster to continue to work. If only 1 server node is running and 2 crashed, then the cluster won't work
s
Yes. Obviously. The database isn't external from the nodes, just outside of k3s. Having an external database means having a database which is not on any of your nodes. For instance, make a MySQL or PostgreSQL database on a separate host and point one or more k3s server nodes at it. It's fully explained in the page I linked to in my first response.
h
Roger. thankyou for the feedback. embedded etcd HA k3s cluster with 4 nodes can only have 1 server fail. While external etcd with 4 nodes can have up to 2 servers fail.... although I don't have all my machines yet(I only have 3) I wonder if using embedded etcd with 3 servers and then having 1 agent, the cluster would tolerate the agent and 1 server failing.... @sticky-summer-13450
s
Yes, it would.
h
@sticky-summer-13450 I really appreciate your feedback. the most important thing I'm testing for: high availability using k3s where I want the most possible nodes/machines to be able to fail and the cluster/all my applications will still be working fine. additionally all the applications in the cluster will be also communicating with eachother, and I want the cluster to run and all the applications to run for a very long time. I think I'm going to test: 1. external etcd DB replicated on 4 control plane nodes (this way, any up to 2 of the control plane nodes can fail and the cluster will still work fine) and 2. embedded etcd DB on 3 control plane nodes with 1 agent node ( this way, the agent can fail and any only 1 of the control plane nodes can fail with the agent and the cluster will still work fine). What I think will happen: First- external etcd DB on 4 control plane nodes would be best if I want up to 2 of any of the 4 nodes to be able to fail and my cluster/apps will still run fine(best node failure cluster tolerance). Second- embedded etcd DB on 3 control plane nodes with 1 agent would allow any but only 1 of the 3 control planes to fail and the agent can fail (less node failure flexibility than using external db with 4 control planes) but this system would perform faster then the first system I mentioned if using the first system can't take the memory and processing requirements of all my apps (embedded etcd is the fall back if the external db method is having trouble with workloads). If the first test method with external DB can handle all my apps fine I will use that system obviously for the benefits of more HA node failure tolerance flexibility. thanks -will I'll also tag @creamy-pencil-82913 to see what he thinks.
c
If only 1 server node is running and 2 crashed, then the cluster won’t work
yes, that is how quorum-based replication and failover works. Doesn’t matter if you’re using embedded etcd or external etcd
You’re not going to get anything more out of managing etcd on your own. the requirements and performance will be identical.
h
@creamy-pencil-82913 roger. embedded it is then.
@creamy-pencil-82913 I created an application and I set "replicas: 2" . So I have 1 pod running in a node and its replica running in another node. I've read thoroughly about replicas but I'm still confused. If a application has 2 replicas are they just the same application? for example would their data outputs be exactly the same? So if one of the replicas fails the application continues to run seamlessly on the other replica. But If I wanted to check the output of 1 applications replica would it be the same as the other? A side note- my application already is configured to persist its data, but in the context of kubernetes would the application still persist data fine, or do I have to setup persistent storage for the applications deployment or create a stateful set. thanks -will
c
they’re just additional copies of the same pod running with the same configuration
How they share state is up to you or the application developer. If they don’t have shared storage, or a common backend, or something else like that, just running more copies of it probably doesn’t do you any good.
Kubernetes is at the end of the day just a very complicated workload scheduler. It doesn’t do anything for you if the application doesn’t natively support running multiple copies.
h
roger that @creamy-pencil-82913 as long as the application(say a sensor) and all its replicas are receiving data from the same source I think your saying that yes, the sensor output from all replicas will be the same because they're all taking in data from the same source. thanks
all the server nodes have embedded etcd....
c
It really depends on the application
replication of the Kubernetes datastore using etcd has nothing to do with replicating the data for your application between copies of it
unless your application happens to store its data in Kubernetes
If I send data to one replica of your application, how does the other replica become aware of that? Do they have a common database backend? are the writing to the same shared filesystem? are the two copies talking directly to each other via some sort of peer channel? I have no idea.
h
roger
well the other replica becomes aware of it because I have a service for the application deployment... the service creates virtual IP that maps to the set of replicas running the application... so the application taking in the sensor data is able to access the sensor app over the service endpoint instead of directly accessing one of the replicas.... the sensor takes data at its service endpoint and kubernetes auto load balances the traffic across all the replicas that are part of the service. this is how I thought this worked..... maybe I'm wrong and need to go over this again @creamy-pencil-82913
c
load-balancing does not usually mean “sends a copy of everything to every backend”
usually it means “sends incoming connections to A SINGLE backend”
that individual back-end is responsible for putting it somewhere that the other replicas are aware of
h
roger
Good morning @creamy-pencil-82913, I hope you are doing well. I enabled 2 replicas from the same application deployment.yaml file to talk to eachother by using shared file system(NFS) by creating a persistent volume claim that specifies the NFS and then mounted that claim to eacah application replica pods. I was also wondering, if all the applications are using MQTT mosqutto broker, couldn't I just create an MQTT image deployment.yaml onto each node, and then configure each of the mqtt replica pods to communicate over the same broker address, and then connect all of the applications that are replicated onto each node to use this same mqtt broker address that is on each node? and then this would solve the problem for example with if the first application on a node fails then the other replicas of that app would pickup exactly where the previous left off because they're all communicating to the same mqtt broker address? I might've worded this wrong but the main idea is if mqtt is deployed and replicated onto each node and they are all using the same mqtt broker address, then all the apps that are also replicated onto each node would use the same mqtt broker address so if a node fails then the same replicated application on a different node would continue to run the same state the first application pod that failed was in... I know this is pretty complex but let me know what you think and if this is possible. thanks -will
If that is wrong I'll just setup the original plan of setting up persistent volume so replicas follow the same state as the original application pod so that if that 1st pod fails then the replicas will continue running the application in the same state that the 1st pod was in
13 Views
Open in Slack
PreviousNext
Powered by