hi the registration script to register nodes hangs fails I n rancher-users #general

Title

big-jordan-45387

04/03/2023, 8:51 PM

hi, the registration script to register nodes hangs/fails, I narrowed down to this command:

curl  --connect-timeout 60 --max-time 60 --write-out "%{http_code}\n"  -sS -H "Authorization: Bearer xxxxxxxxx" -H "X-Cattle-Id: yyyyyyyyyyy" -H "X-Cattle-Role-Etcd: true" -H "X-Cattle-Role-Control-Plane: true" -H "X-Cattle-Role-Worker: true" -H "X-Cattle-Node-Name: " -H "X-Cattle-Address: " -H "X-Cattle-Internal-Address: " -H "X-Cattle-Labels: <http://cattle.io/os=linux|cattle.io/os=linux>" -H "X-Cattle-Taints: " "<https://zzzzzzzzzzz>"/v3/connect/agent -o /var/lib/rancher/agent/rancher2_connection_info.json

The command above times out and the script ends up in a loop. Has anyone any idea of how can I do next to understand what is going on?

creamy-pencil-82913

04/03/2023, 9:12 PM

Basic network troubleshooting is where i’d start. Can you resolve the address? can you ping it? Can you perform a basic curl of that address without the extra parameters?

big-jordan-45387

04/03/2023, 9:13 PM

yeah, I don't think it is network issue since the node did download the script over the network from the rancher server

also I get an 401 http error running the curl command without the X-Cattle_Id header

if I run curl in verbose mode, I can see the traffic, both client and server agrees on the https cert ... then it hangs

and I can see traffic going both ways when I run tcpdump

creamy-pencil-82913

04/03/2023, 9:20 PM

are you sure that curl command is actually where it’s hanging then, and not actually some command after that?

big-jordan-45387

04/03/2023, 9:21 PM

I put "echo" everywhere to trace the script execution and that curl command is where it hangs

now that you ask, let me double check

so, as mentioned, I have the script like this

echo "3"

            echo "######## ATTEMPT $i of $RETRYCOUNT"

            echo "calling ===> curl $noproxy --connect-timeout 60 --max-time 60 --write-out \"%{http_code}\n\" ${CURL_CAFLAG} ${CURL_LOG} -H \"Authorization: Bearer ${CATTLE_TOKEN}\" -H \"X-Cattle-Id: ${CATTLE_ID}\" -H \"X-Cattle-Role-Etcd: ${CATTLE_ROLE_ETCD}\" -H \"X-Cattle-Role-Control-Plane: ${CATTLE_ROLE_CONTROLPLANE}\" -H \"X-Cattle-Role-Worker: ${CATTLE_ROLE_WORKER}\" -H \"X-Cattle-Node-Name: ${CATTLE_NODE_NAME}\" -H \"X-Cattle-Address: ${CATTLE_ADDRESS}\" -H \"X-Cattle-Internal-Address: ${CATTLE_INTERNAL_ADDRESS}\" -H \"X-Cattle-Labels: ${CATTLE_LABELS}\" -H \"X-Cattle-Taints: ${CATTLE_TAINTS}\" \"${CATTLE_SERVER}\"/v3/connect/agent -o ${CATTLE_AGENT_VAR_DIR}/rancher2_connection_info.json"

            RESPONSE=$(curl $noproxy --connect-timeout 60 --max-time 60 --write-out "%{http_code}\n" ${CURL_CAFLAG} ${CURL_LOG} -H "Authorization: Bearer ${CATTLE_TOKEN}" -H "X-Cattle-Id: ${CATTLE_ID}" -H "X-Cattle-Role-Etcd: ${CATTLE_ROLE_ETCD}" -H "X-Cattle-Role-Control-Plane: ${CATTLE_ROLE_CONTROLPLANE}" -H "X-Cattle-Role-Worker: ${CATTLE_ROLE_WORKER}" -H "X-Cattle-Node-Name: ${CATTLE_NODE_NAME}" -H "X-Cattle-Address: ${CATTLE_ADDRESS}" -H "X-Cattle-Internal-Address: ${CATTLE_INTERNAL_ADDRESS}" -H "X-Cattle-Labels: ${CATTLE_LABELS}" -H "X-Cattle-Taints: ${CATTLE_TAINTS}" "${CATTLE_SERVER}"/v3/connect/agent -o ${CATTLE_AGENT_VAR_DIR}/rancher2_connection_info.json)

            echo "call done with response $RESPONSE"

and I get the output

3
######## ATTEMPT 1 of 4500
calling ===> curl  --connect-timeout 60 --max-time 60 --write-out "%{http_code}\n"  -sS -H "Authorization: Bearer ..." -H "X-Cattle-Id: ..." -H "X-Cattle-Role-Etcd: true" -H "X-Cattle-Role-Control-Plane: true" -H "X-Cattle-Role-Worker: true" -H "X-Cattle-Node-Name: " -H "X-Cattle-Address: " -H "X-Cattle-Internal-Address: " -H "X-Cattle-Labels: <http://cattle.io/os=linux|cattle.io/os=linux>" -H "X-Cattle-Taints: " "https://..."/v3/connect/agent -o /var/lib/rancher/agent/rancher2_connection_info.json

and I manually call the script like

./system-agent-install.sh --server https://... --label '<http://cattle.io/os=linux|cattle.io/os=linux>' --token .... --etcd --controlplane --worker

im pretty sure the curl command hangs --> the rancher server is not sending a response back

hundreds-evening-84071

04/03/2023, 9:28 PM

DNS? If its curl command, can you resolve the name?

big-jordan-45387

04/03/2023, 9:28 PM

yes can resolve the name

hundreds-evening-84071

04/03/2023, 9:30 PM

nc -v server 443 ?

big-jordan-45387

04/03/2023, 9:33 PM

if I do curl to the rancher server I get a json back

network is fine

requests with authorization bearer and x-cattle-id headers hangs (at least to the /v3/connect/agent endpoint)

creamy-pencil-82913

04/03/2023, 9:34 PM

maybe check logs on the rancher pod then?

big-jordan-45387

04/03/2023, 9:35 PM

is this rancher pod running on the rke2 machine or is it running in the hosts running rancher?

creamy-pencil-82913

04/03/2023, 9:48 PM

the one that you’re making the request against

big-jordan-45387

04/03/2023, 9:49 PM

ah, ok I dont have access to that host, I will ask for help to the rancher sys admins tomorrow

4 Views

#general Join Slack