SSH Operations Runbook¶

Operational procedures for SSH key management - Step-by-step guides for common SSH operations.

Table of Contents¶

Overview
Generate New SSH Key
Add Key to Servers
Remove Key from Servers
Rotate SSH Keys
Troubleshooting
Emergency Access

Overview¶

Purpose¶

This runbook provides step-by-step procedures for: - Generating SSH keys for new devices - Adding/removing keys from servers - Rotating keys periodically - Troubleshooting SSH issues - Emergency access scenarios

Prerequisites¶

Physical access to device (MacBook, iPad, PC)
Existing SSH access to servers (for adding keys)
OR admin assistance (for first-time setup)

Generate New SSH Key¶

When to Use¶

New device (new MacBook, iPad, or PC)
Compromised key (suspected or confirmed)
Periodic rotation (recommended yearly)

Procedure¶

Step 1: Choose Key Name

Follow the device-specific naming convention:

Device Type	Key Name
MacBook	`id_ed25519_macbook`
iPad	`id_ed25519_ipad`
PC (Windows/Linux)	`id_ed25519_pc`
Secondary MacBook	`id_ed25519_macbook2`

Step 2: Generate Key

# On device (local terminal, not server)
ssh-keygen -t ed25519 -C "kavi@<device>" -f ~/.ssh/id_ed25519_<device>

Examples:

# MacBook
ssh-keygen -t ed25519 -C "kavi@macbook" -f ~/.ssh/id_ed25519_macbook

# iPad (using a-Shell or other terminal app)
ssh-keygen -t ed25519 -C "kavi@ipad" -f ~/.ssh/id_ed25519_ipad

# PC
ssh-keygen -t ed25519 -C "kavi@pc" -f ~/.ssh/id_ed25519_pc

Prompts:

Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):

Passphrase recommendation: - ✅ Use passphrase for enhanced security (if device supports secure storage) - ⚠️ No passphrase acceptable for devices with full-disk encryption (FDE)

Step 3: Verify Generation

# List generated keys
ls -la ~/.ssh/id_ed25519_<device>*

# Expected output:
# -rw-------  1 kavi  staff  411 Jan 15 10:00 id_ed25519_macbook       ← Private key
# -rw-r--r--  1 kavi  staff  100 Jan 15 10:00 id_ed25519_macbook.pub   ← Public key

Check permissions: - Private key: 600 (-rw-------) - Public key: 644 (-rw-r--r--)

If permissions are wrong:

chmod 600 ~/.ssh/id_ed25519_macbook
chmod 644 ~/.ssh/id_ed25519_macbook.pub

Step 4: View Public Key

# Display public key (this is safe to share)
cat ~/.ssh/id_ed25519_macbook.pub

# Example output:
# ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAbC... kavi@macbook

Copy this value - you'll need it for the next section.

Add Key to Servers¶

When to Use¶

New device needs access to servers
New server added to infrastructure
Restoring access after key removal

Option A: Using Existing SSH Access¶

If you have SSH access from another device:

Step 1: Copy Public Key

# On device with new key
cat ~/.ssh/id_ed25519_macbook.pub
# Copy the output

Step 2: Add to Hetzner VPS

# SSH from device that already has access
ssh kavi@100.80.53.55

# On server
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAbC... kavi@macbook" >> ~/.ssh/authorized_keys

# Verify unique (no duplicates)
sort ~/.ssh/authorized_keys | uniq > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

# Set correct permissions
chmod 600 ~/.ssh/authorized_keys

# Verify
cat ~/.ssh/authorized_keys

Step 3: Add to Kimsufi Server

# SSH from device that already has access
ssh ubuntu@100.81.231.36

# On server (same process as Hetzner)
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAbC... kavi@macbook" >> ~/.ssh/authorized_keys
sort ~/.ssh/authorized_keys | uniq > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Step 4: Test from New Device

# On new device
ssh kavi@100.80.53.55

# Should connect without password

Option B: Using ssh-copy-id¶

If ssh-copy-id is available:

# From new device
ssh-copy-id -i ~/.ssh/id_ed25519_macbook.pub kavi@100.80.53.55
ssh-copy-id -i ~/.ssh/id_ed25519_macbook.pub ubuntu@100.81.231.36

# This automatically adds key to ~/.ssh/authorized_keys

Note: Requires existing password-based or key-based SSH access.

Option C: Via Admin/Another User¶

If you don't have any access (first device):

Send public key to admin:

# On your device
cat ~/.ssh/id_ed25519_macbook.pub
# Copy and send to admin via secure channel (Signal, email)

Admin adds to servers:

# Admin runs on each server
echo "<your-public-key>" >> ~kavi/.ssh/authorized_keys
# (or ~/.ssh/authorized_keys for user ubuntu on Kimsufi)

Test connection:
```
ssh kavi@100.80.53.55
```

Remove Key from Servers¶

When to Use¶

Device lost or stolen
Device decommissioned (old laptop)
Key compromised
User offboarding

Procedure¶

Step 1: Identify Key to Remove

# On server
cat ~/.ssh/authorized_keys

# Find the key with matching comment, e.g.:
# ssh-ed25519 AAAAC3Nza... kavi@old-macbook  ← Remove this one

Step 2: Remove Key

Option A: Edit File Manually

# On server
nano ~/.ssh/authorized_keys
# or
vim ~/.ssh/authorized_keys

# Delete the line containing the key
# Save and exit

Option B: Use grep to Filter

# On server
# Remove key with specific comment
grep -v "kavi@old-macbook" ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

# Verify
cat ~/.ssh/authorized_keys

Step 3: Repeat for All Servers

# Hetzner VPS
ssh kavi@100.80.53.55
grep -v "kavi@old-macbook" ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

# Kimsufi
ssh ubuntu@100.81.231.36
grep -v "kavi@old-macbook" ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

Step 4: Verify Removal

# Try to connect from old device (should fail)
ssh kavi@100.80.53.55
# Expected: Permission denied (publickey)

Rotate SSH Keys¶

When to Use¶

Annual security practice (recommended)
After security incident
When device is replaced

Procedure¶

Overview: Generate new key → Add new key → Remove old key

Step 1: Generate New Key

# On device
ssh-keygen -t ed25519 -C "kavi@macbook-2025" -f ~/.ssh/id_ed25519_macbook_2025

Step 2: Add New Key to Servers

# Add to Hetzner
ssh kavi@100.80.53.55
echo "ssh-ed25519 AAAAC3... kavi@macbook-2025" >> ~/.ssh/authorized_keys

# Add to Kimsufi
ssh ubuntu@100.81.231.36
echo "ssh-ed25519 AAAAC3... kavi@macbook-2025" >> ~/.ssh/authorized_keys

Step 3: Update SSH Config

# On device
nano ~/.ssh/config

# Change:
# IdentityFile ~/.ssh/id_ed25519_macbook
# To:
# IdentityFile ~/.ssh/id_ed25519_macbook_2025

Step 4: Test New Key

# Verify connection works with new key
ssh kavi@100.80.53.55
ssh ubuntu@100.81.231.36

Step 5: Remove Old Key from Servers

# On Hetzner
ssh kavi@100.80.53.55
grep -v "kavi@macbook" ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

# On Kimsufi
ssh ubuntu@100.81.231.36
grep -v "kavi@macbook" ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

Step 6: Archive Old Key

# On device
mv ~/.ssh/id_ed25519_macbook ~/.ssh/archived/id_ed25519_macbook.old
mv ~/.ssh/id_ed25519_macbook.pub ~/.ssh/archived/id_ed25519_macbook.pub.old

Troubleshooting¶

"Permission denied (publickey)"¶

Symptom:

kavi@macbook ~ % ssh kavi@100.80.53.55
kavi@100.80.53.55: Permission denied (publickey).

Causes: 1. Public key not in server's authorized_keys 2. Wrong private key being used 3. File permissions incorrect 4. SSH config using wrong IdentityFile

Solution:

Check 1: Public key on server?

# On server (SSH from another device)
cat ~/.ssh/authorized_keys | grep "kavi@macbook"

# If missing, add it (see "Add Key to Servers" above)

Check 2: Correct private key?

# On device
ls -la ~/.ssh/id_ed25519_*

# Verify you're using the right key
cat ~/.ssh/config
# Should have:
# IdentityFile ~/.ssh/id_ed25519_macbook

Check 3: File permissions

# On device
ls -la ~/.ssh/id_ed25519_macbook
# Should be: -rw------- (600)

# Fix if wrong:
chmod 600 ~/.ssh/id_ed25519_macbook

Check 4: SSH agent

# On device
ssh-add -l
# Lists loaded keys

# If your key is not listed:
ssh-add ~/.ssh/id_ed25519_macbook

Check 5: Verbose SSH

# Debug connection
ssh -v kavi@100.80.53.55

# Look for:
# debug1: Offering public key: ~/.ssh/id_ed25519_macbook ED25519 ...
# debug1: Server accepts key: ... ← Should see this
# debug1: Authentication succeeded (publickey). ← Success

"Too many authentication failures"¶

Symptom:

Received disconnect from 100.80.53.55: Too many authentication failures

Cause: SSH trying too many keys (server limit is usually 5-6)

Solution:

Option A: Specify key explicitly

ssh -i ~/.ssh/id_ed25519_macbook kavi@100.80.53.55

Option B: Update SSH config

# ~/.ssh/config
Host hetzner
    HostName 100.80.53.55
    User kavi
    IdentityFile ~/.ssh/id_ed25519_macbook
    IdentitiesOnly yes  ← Add this

"WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!"¶

Symptom:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Causes: 1. Server was reinstalled (legitimate) 2. Server IP reassigned to different machine 3. Man-in-the-middle attack (rare)

Solution:

If you reinstalled the server (legitimate):

# Remove old host key
ssh-keygen -R 100.80.53.55

# Or edit known_hosts
nano ~/.ssh/known_hosts
# Delete the line for 100.80.53.55

# Reconnect (will prompt to accept new key)
ssh kavi@100.80.53.55

If you didn't reinstall (investigate!): - Contact admin - Verify server fingerprint out-of-band - Do NOT proceed if unsure

Key file permissions too open¶

Symptom:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '~/.ssh/id_ed25519_macbook' are too open.

Cause: Private key is readable by others (security risk)

Solution:

chmod 600 ~/.ssh/id_ed25519_macbook

Emergency Access¶

Scenario 1: Lost All SSH Keys¶

Problem: All devices lost/stolen, no SSH access

Solution: Physical or console access required

Option A: Hetzner Rescue System

Log into Hetzner Robot: https://robot.hetzner.com
Activate Rescue System for VPS
Reboot into rescue mode
Access via VNC console
Mount filesystem
Add new SSH key to /mnt/home/kavi/.ssh/authorized_keys
Reboot normally
SSH with new key

Option B: Cloud Console (if available)

Log into cloud provider dashboard
Use web console / serial console
Login with password (if enabled) or single-use password
Add new SSH key
Exit console
SSH with new key

Scenario 2: Locked Out of Server¶

Problem: authorized_keys corrupted or accidentally deleted

Solution: Same as Scenario 1 - requires physical/console access

Prevention: - Keep backup key on secondary device - Enable password authentication as fallback (less secure) - Maintain console access credentials

Scenario 3: Key Compromised¶

Problem: Private key stolen or exposed

Immediate actions:

Remove compromised key from ALL servers:

# On each server
ssh kavi@100.80.53.55
grep -v "kavi@compromised-device" ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.tmp
mv ~/.ssh/authorized_keys.tmp ~/.ssh/authorized_keys

Generate new key:

ssh-keygen -t ed25519 -C "kavi@new-device" -f ~/.ssh/id_ed25519_new

Add new key to servers (using remaining access)

Audit server logs for unauthorized access:

# Check SSH logs
sudo grep "Accepted publickey" /var/log/auth.log
sudo grep "kavi" /var/log/auth.log | grep "session opened"

Rotate secrets in Infisical (if server access was compromised)

Summary¶

SSH Operations: - ✅ Generate keys: ssh-keygen -t ed25519 -C "kavi@device" -f ~/.ssh/id_ed25519_device - ✅ Add to servers: echo "<public-key>" >> ~/.ssh/authorized_keys - ✅ Remove from servers: grep -v "comment" authorized_keys > authorized_keys.tmp - ✅ Rotate annually: Generate new → Add new → Remove old - ✅ Troubleshoot: Check permissions, SSH config, verbose mode

Key Principles: 1. One key per device (never share) 2. Public key safe to share, private key never shared 3. File permissions: Private 600, Public 644 4. Use descriptive comments: kavi@macbook, kavi@ipad 5. Keep backup keys on secondary devices

Emergency: - Physical/console access if all keys lost - Immediate removal if key compromised - Audit logs after security incident

What's Next: - See SSH Keys Architecture - detailed SSH strategy - See Secrets Management - Infisical operations - See Quick Start - first-time setup