I know the HA setup But if I have 4 nodes 3 servers + 1 agen rancher-users #k3s

Title

# k3s

handsome-autumn-77266

02/27/2023, 4:08 PM

I know the HA setup. But if I have 4 nodes(3 servers + 1 agent) in total: I would love to be able to shut off 3 of them and have only 1 running and my applications to all still be running normally even if a bit slower. But there's no way to do this with HA obviously. I was thinking the other day, couldn't I just create individual clusters on each machine, and have my applications running on all of them? and that way I could be able to shut off 3 machines and my applications technically would still be running?

sticky-summer-13450

02/28/2023, 9:46 AM

Have you thought of having an external database? I believe, and I'm sure someone will correct me if I'm wrong, but the requirement for an odd number of servers is only with using etcd for the database. https://docs.k3s.io/installation/ha

handsome-autumn-77266

02/28/2023, 10:22 PM

Yes @sticky-summer-13450 but what if the external database fails? thats still a single point of failure right?

sticky-summer-13450

02/28/2023, 10:37 PM

You want to scale down to 1 master and you want HA - I don't think kubernetes does "cake and eat it" 😃 I gave you a solution to your request, moving the goalposts isn't a fair response.

handsome-autumn-77266

02/28/2023, 10:52 PM

roger. I think I figured it out. thanks for the input.

👍 1

@sticky-summer-13450 so I created 3 server nodes and 1 agent node in k3s cluster. external etcd DB was replicated onto each of these nodes. but this system still requires at all times that at least 2 server nodes need to be running for the cluster to continue to work. If only 1 server node is running and 2 crashed, then the cluster won't work

sticky-summer-13450

03/01/2023, 7:54 AM

Yes. Obviously. The database isn't external from the nodes, just outside of k3s. Having an external database means having a database which is not on any of your nodes. For instance, make a MySQL or PostgreSQL database on a separate host and point one or more k3s server nodes at it. It's fully explained in the page I linked to in my first response.

handsome-autumn-77266

03/01/2023, 4:00 PM

Roger. thankyou for the feedback. embedded etcd HA k3s cluster with 4 nodes can only have 1 server fail. While external etcd with 4 nodes can have up to 2 servers fail.... although I don't have all my machines yet(I only have 3) I wonder if using embedded etcd with 3 servers and then having 1 agent, the cluster would tolerate the agent and 1 server failing.... @sticky-summer-13450

sticky-summer-13450

03/01/2023, 6:33 PM

Yes, it would.

handsome-autumn-77266

03/01/2023, 9:02 PM

@sticky-summer-13450 I really appreciate your feedback. the most important thing I'm testing for: high availability using k3s where I want the most possible nodes/machines to be able to fail and the cluster/all my applications will still be working fine. additionally all the applications in the cluster will be also communicating with eachother, and I want the cluster to run and all the applications to run for a very long time. I think I'm going to test: 1. external etcd DB replicated on 4 control plane nodes (this way, any up to 2 of the control plane nodes can fail and the cluster will still work fine) and 2. embedded etcd DB on 3 control plane nodes with 1 agent node ( this way, the agent can fail and any only 1 of the control plane nodes can fail with the agent and the cluster will still work fine). What I think will happen: First- external etcd DB on 4 control plane nodes would be best if I want up to 2 of any of the 4 nodes to be able to fail and my cluster/apps will still run fine(best node failure cluster tolerance). Second- embedded etcd DB on 3 control plane nodes with 1 agent would allow any but only 1 of the 3 control planes to fail and the agent can fail (less node failure flexibility than using external db with 4 control planes) but this system would perform faster then the first system I mentioned if using the first system can't take the memory and processing requirements of all my apps (embedded etcd is the fall back if the external db method is having trouble with workloads). If the first test method with external DB can handle all my apps fine I will use that system obviously for the benefits of more HA node failure tolerance flexibility. thanks -will I'll also tag @creamy-pencil-82913 to see what he thinks.

creamy-pencil-82913

03/01/2023, 9:11 PM

If only 1 server node is running and 2 crashed, then the cluster won’t work

yes, that is how quorum-based replication and failover works. Doesn’t matter if you’re using embedded etcd or external etcd

Have you read https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance

You’re not going to get anything more out of managing etcd on your own. the requirements and performance will be identical.

handsome-autumn-77266

03/01/2023, 9:26 PM

@creamy-pencil-82913 roger. embedded it is then.

@creamy-pencil-82913 I created an application and I set "replicas: 2" . So I have 1 pod running in a node and its replica running in another node. I've read thoroughly about replicas but I'm still confused. If a application has 2 replicas are they just the same application? for example would their data outputs be exactly the same? So if one of the replicas fails the application continues to run seamlessly on the other replica. But If I wanted to check the output of 1 applications replica would it be the same as the other? A side note- my application already is configured to persist its data, but in the context of kubernetes would the application still persist data fine, or do I have to setup persistent storage for the applications deployment or create a stateful set. thanks -will

creamy-pencil-82913

03/09/2023, 10:07 PM

they’re just additional copies of the same pod running with the same configuration

How they share state is up to you or the application developer. If they don’t have shared storage, or a common backend, or something else like that, just running more copies of it probably doesn’t do you any good.

Kubernetes is at the end of the day just a very complicated workload scheduler. It doesn’t do anything for you if the application doesn’t natively support running multiple copies.

handsome-autumn-77266

03/09/2023, 11:31 PM

roger that @creamy-pencil-82913 as long as the application(say a sensor) and all its replicas are receiving data from the same source I think your saying that yes, the sensor output from all replicas will be the same because they're all taking in data from the same source. thanks

all the server nodes have embedded etcd....

creamy-pencil-82913

03/09/2023, 11:48 PM

It really depends on the application

replication of the Kubernetes datastore using etcd has nothing to do with replicating the data for your application between copies of it

unless your application happens to store its data in Kubernetes

If I send data to one replica of your application, how does the other replica become aware of that? Do they have a common database backend? are the writing to the same shared filesystem? are the two copies talking directly to each other via some sort of peer channel? I have no idea.

handsome-autumn-77266

03/09/2023, 11:53 PM

roger

well the other replica becomes aware of it because I have a service for the application deployment... the service creates virtual IP that maps to the set of replicas running the application... so the application taking in the sensor data is able to access the sensor app over the service endpoint instead of directly accessing one of the replicas.... the sensor takes data at its service endpoint and kubernetes auto load balances the traffic across all the replicas that are part of the service. this is how I thought this worked..... maybe I'm wrong and need to go over this again @creamy-pencil-82913

creamy-pencil-82913

03/10/2023, 12:09 AM

load-balancing does not usually mean “sends a copy of everything to every backend”

usually it means “sends incoming connections to A SINGLE backend”

that individual back-end is responsible for putting it somewhere that the other replicas are aware of

handsome-autumn-77266

03/10/2023, 12:11 AM

roger

Good morning @creamy-pencil-82913, I hope you are doing well. I enabled 2 replicas from the same application deployment.yaml file to talk to eachother by using shared file system(NFS) by creating a persistent volume claim that specifies the NFS and then mounted that claim to eacah application replica pods. I was also wondering, if all the applications are using MQTT mosqutto broker, couldn't I just create an MQTT image deployment.yaml onto each node, and then configure each of the mqtt replica pods to communicate over the same broker address, and then connect all of the applications that are replicated onto each node to use this same mqtt broker address that is on each node? and then this would solve the problem for example with if the first application on a node fails then the other replicas of that app would pickup exactly where the previous left off because they're all communicating to the same mqtt broker address? I might've worded this wrong but the main idea is if mqtt is deployed and replicated onto each node and they are all using the same mqtt broker address, then all the apps that are also replicated onto each node would use the same mqtt broker address so if a node fails then the same replicated application on a different node would continue to run the same state the first application pod that failed was in... I know this is pretty complex but let me know what you think and if this is possible. thanks -will

If that is wrong I'll just setup the original plan of setting up persistent volume so replicas follow the same state as the original application pod so that if that 1st pod fails then the replicas will continue running the application in the same state that the 1st pod was in

6 Views