So... I decided to run my own Solana RPC node. You know, for that sweet independence from rate-limited public endpoints, faster queries, and honestly? Just to say I did it. How hard could it be, right?
Spoiler: It's not that hard, but man, there are some gotchas that'll eat hours of your time if you don't know about them upfront. Let me save you some pain.
Why Even Bother?
Before we dive in, let's be real about why you'd want to do this. Public RPC endpoints are great until they're not. Rate limits. Throttling. That one critical transaction that times out because everyone and their dog is hammering the same endpoint during an NFT mint.
Running your own node gives you:
No rate limits (well, except your own hardware)
Lower latency if you're hosting close to your application
Full control over configurations
The ability to query historical data without restrictions
But here's the thing – it's not cheap and it's definitely not "set it and forget it." You need beefy hardware (we're talking 256GB+ RAM, fast NVMe drives, multiple TB of storage) and patience. Lots of patience.
The Docker Trap (Don't Do What I Did First)
Okay, so naturally, my first instinct was Docker. I mean, everything's better in Docker, right? Containerize all the things!
Wrong. Well, kind of wrong.
I spent hours getting a Docker setup working, and it technically ran, but here's what nobody tells you upfront: Docker is not recommended for Solana validators or RPC nodes. Like, officially not recommended. The performance overhead from containerization matters when you're processing hundreds of thousands of transactions per second.
The Agave documentation basically says "don't use Docker for production" and, to be honest, I should've read that more carefully before diving in. But we live and learn, I guess.
Going Native: Agave Validator
Alright, so native it is. Solana – actually, let me correct that – Agave (that's what we call it now since the Anza fork) needs to run directly on Ubuntu. No containers. Just you, Linux, and a whole lot of system tuning.
Here's the thing: the installation itself is actually pretty straightforward. You run one command:
sh -c "$(curl -sSfL https://release.anza.xyz/stable/install)"
Boom. Agave is installed. Add it to your PATH and you're technically ready to go.
But – and this is a big but – you're not really ready. Not even close.
The Snapshot Download Marathon
Here's where my expectations got... recalibrated. When you first start an RPC node, it needs to download a snapshot of the current blockchain state. This snapshot is about 100GB.
"100GB, okay, that'll take like an hour on my connection," I thought.
Three hours later, I was still watching progress logs tick by at 2.3%. The download speed fluctuates wildly (anywhere from 8MB/s to 18MB/s in my case), and you're at the mercy of whoever's serving you the snapshot.
Actually, wait – let me give you a pro tip here. You can download snapshots from different sources. I tried:
Just... budget time for this. Like, a lot of time. And then after downloading, it needs to unpack. That's another 20-40 minutes. And then it needs to catch up to the network tip.
First-time setup? You're looking at 4-6 hours minimum. Maybe more.
Configuration Hell (Or: That One Character That Broke Everything)
So I got the validator running, logs are streaming, everything looks good. Except... blocks weren't being stored. Like, at all.
The RPC would sync, process blocks, show healthy metrics, but when I tried to query a block that happened 5 minutes ago? "Block not available."
I spent way too long debugging this. Here's what I learned:
1. The --limit-ledger-size flag is critical
If you just write --limit-ledger-size without a number, the validator will aggressively prune everything. Like, immediately. You need to specify: --limit-ledger-size 50000000 (that's 50 million shreds, gives you a few days of data).
2. You need --enable-extended-tx-metadata-storage
Without this flag, transaction history just... doesn't get stored properly. It's not enough to just have --enable-rpc-transaction-history. You need both.
3. Validator pubkeys matter
This one almost killed me. I had a typo in one of the --known-validator keys. One extra character. DE1bawNcRJB9rVm3buyMVfr4mFejNKTyJb4pnB2LLz9fF instead of DE1bawNcRJB9rVm3buyMVfr4mFejNKTyJb4pnB2LLz9f.
The validator wouldn't start. Error logs just said "String is the wrong size." Took me forever to spot that extra 'F' at the end. Solana pubkeys are exactly 44 characters. Not 43. Not 45. Exactly 44.
System Tuning: This Actually Matters
Here's something I didn't appreciate at first: system tuning isn't optional. It's not like "oh, nice to have for a bit more performance." No, your node will struggle or fail without it.
You need to configure:
UDP buffer sizes (Solana uses a LOT of UDP traffic)
File descriptor limits (it opens thousands of files)
Memory mapped files limits
TCP parameters
VM settings
I'm not gonna lie, I copy-pasted most of this from community guides. But it works. You dump it all in /etc/sysctl.d/21-agave-validator.conf and apply it. The difference is night and day.
The Blockstore Mystery
Okay, so even with proper config, I had this issue where rocksdb – that's where blocks are actually stored – just wasn't being created. The ledger directory would have accounts and snapshots but no rocksdb folder.
Turns out, if you started with bad configuration and then fixed it, sometimes you need to nuke everything and start fresh. Like, literally delete the ledger and accounts directories and re-download the snapshot.
It's not ideal, but it's the nuclear option that actually works. And honestly? After trying to debug for hours, sometimes you just need to start clean.
Monitoring: How Do You Even Know It's Working?
Once it's running, how do you know it's actually working? A few key things:
Check the slot:
solana slot
Compare it to mainnet:
curl -s https://api.mainnet-beta.solana.com -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1, "method":"getSlot"}'
If you're within 100 slots of mainnet, you're synced. If you're thousands behind... well, you're catching up. Or you're not catching up and need to restart. RPC nodes can take hours to catch up, and sometimes they just don't.
Check if rocksdb exists:
ls -lh /mnt/data/solana/ledger/rocksdb/
If that directory doesn't exist, your blocks aren't being stored. Period.
Try to query a block:
SLOT=$(solana slot)
TEST=$((SLOT - 300))
curl http://localhost:8899 -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","id":1,"method":"getBlock","params":['$TEST',{"encoding":"json"}]}'
If you get back transaction data instead of "Block not available," you're golden.
The Reality Check
Let me be honest here. Running your own Solana RPC node is kind of a pain. It's resource-intensive, it takes time to set up properly, and it requires ongoing maintenance.
But here's the thing – once it's running, once you've got it properly configured and synced, it's nice. No rate limits. Fast queries. Full control.
Is it worth it? Depends on your use case. If you're building something serious on Solana and public endpoints are becoming a bottleneck, yeah, absolutely. If you're just experimenting or doing occasional queries, probably not.
The hardware requirements alone (I'm running on a machine with 96 cores, 1.2TB RAM, and 14TB NVMe storage) put this out of reach for casual use. This is a professional-grade setup.
My Final Setup
After all that trial and error, here's what actually works for me:
Native Agave installation (no Docker)
Systemd service for management
Proper system tuning applied
50 million shred limit for ledger size
Extended transaction metadata enabled
Log rotation configured (logs fill up fast)
Known validator list properly configured
The validator runs as a systemd service, auto-starts on boot, and just... works. I can query blocks from the last few days, transaction history works, and it keeps up with the network.
Would I Do It Again?
Yeah, I think so. But I'd do it differently. I'd skip the Docker phase entirely. I'd budget more time for the initial sync. I'd pay more attention to the known-validator keys (seriously, double-check those).
And honestly? I'd probably start with a smaller setup first, maybe on testnet, just to understand the process before committing hardware to mainnet.
Running a Solana RPC node isn't something you do on a whim. But if you need it, if you're building something that requires it, the control and performance you get make the hassle worth it.
Just... maybe grab a coffee. Or three. You're gonna be watching progress bars for a while.
Complete Technical Setup Guide
Alright, enough story time. Here's the step-by-step guide with every command you need.
Prerequisites
1. System Preparation
Install Dependencies
sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential pkg-config libssl-dev libudev-dev zstd curl git bc
System Tuning (CRITICAL)
sudo tee /etc/sysctl.d/21-agave-validator.conf > /dev/null << 'EOF'
net.ipv4.tcp_rmem=10240 87380 12582912
net.ipv4.tcp_wmem=10240 87380 12582912
net.ipv4.tcp_congestion_control=westwood
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=1
net.ipv4.tcp_low_latency=1
net.core.rmem_max=134217728
net.core.rmem_default=134217728
net.core.wmem_max=134217728
net.core.wmem_default=134217728
kernel.timer_migration=0
kernel.hung_task_timeout_secs=30
kernel.pid_max=49152
vm.swappiness=30
vm.max_map_count=1000000
vm.stat_interval=10
vm.dirty_ratio=40
vm.dirty_background_ratio=10
vm.min_free_kbytes=3000000
fs.nr_open=1000000
EOF
sudo sysctl -p /etc/sysctl.d/21-agave-validator.conf
File Descriptor Limits
sudo tee /etc/security/limits.d/90-agave-nofiles.conf > /dev/null << 'EOF'
* - nofile 1000000
* - memlock 2000000
EOF
Setup Firewall (Optional)
sudo apt install ufw -y
sudo ufw allow 22/tcp
sudo ufw allow 8000:8020/udp
sudo ufw allow 8000:8020/tcp
sudo ufw allow 8899/tcp
sudo ufw enable
2. Install Agave
sh -c "$(curl -sSfL https://release.anza.xyz/stable/install)"
export PATH="$HOME/.local/share/solana/install/active_release/bin:$PATH"
echo 'export PATH="$HOME/.local/share/solana/install/active_release/bin:$PATH"' >> ~/.bashrc
agave-validator --version
solana --version
3. Create Directories
sudo mkdir -p /mnt/data/solana/ledger
sudo mkdir -p /mnt/data/solana/accounts
mkdir -p ~/log
sudo chown -R $USER:$USER /mnt/data/solana/
4. Generate Keypairs
solana-keygen new -o ~/validator-keypair.json --no-bip39-passphrase
solana config set --keypair ~/validator-keypair.json
solana-keygen new -o ~/vote-account-keypair.json --no-bip39-passphrase
solana-keygen pubkey ~/validator-keypair.json
5. Download Snapshots (Optional but Recommended)
Option A: Let Validator Download Automatically
Skip this section - the validator will download snapshots on first start.
Option B: Pre-download Snapshots (Faster)
cd /tmp
wget -c https://snapshot.latitude.sh/mainnet-beta/snapshot.tar.zst
wget -c https://snapshot.latitude.sh/mainnet-beta/incremental-snapshot.tar.zst
echo "Extracting snapshots... This takes 20-40 minutes"
zstd -d snapshot.tar.zst --stdout | tar -xf - -C /mnt/data/solana/ledger/
zstd -d incremental-snapshot.tar.zst --stdout | tar -xf - -C /mnt/data/solana/ledger/
rm -f *.tar.zst
6. Create Validator Startup Script
cat > ~/start-validator.sh << 'EOF'
PATH=$HOME/.local/share/solana/install/active_release/bin:/usr/sbin:/usr/bin:/sbin:/bin
export RUST_BACKTRACE=1
export RUST_LOG=solana=info
exec agave-validator \
--identity ~/validator-keypair.json \
--entrypoint entrypoint.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
--rpc-port 8899 \
--dynamic-port-range 8000-8020 \
--gossip-port 8001 \
--no-voting \
--private-rpc \
--rpc-bind-address 0.0.0.0 \
--enable-rpc-transaction-history \
--enable-extended-tx-metadata-storage \
--wal-recovery-mode skip_any_corrupted_record \
--vote-account ~/vote-account-keypair.json \
--log ~/log/agave-validator.log \
--accounts /mnt/data/solana/accounts \
--ledger /mnt/data/solana/ledger \
--limit-ledger-size 50000000 \
--full-rpc-api \
--known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
--known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
--known-validator DE1bawNcRJB9rVm3buyMVfr4mFejNKTyJb4pnB2LLz9f \
--known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
--expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d
EOF
chmod +x ~/start-validator.sh
Key Configuration Flags Explained:
--limit-ledger-size 50000000: Keep ~50M shreds (2-3 days of data)
--enable-rpc-transaction-history: Enable transaction history queries
--enable-extended-tx-metadata-storage: Store full transaction metadata
--private-rpc: Don't advertise RPC to the network
--no-voting: Run as RPC only (not a validator)
--full-rpc-api: Enable all RPC methods
7. Setup Systemd Service
USER_NAME=$(whoami)
USER_HOME=$(eval echo ~$USER_NAME)
sudo tee /etc/systemd/system/agave-validator.service > /dev/null << EOF
[Unit]
Description=Agave Validator
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=on-failure
RestartSec=1
LimitNOFILE=1000000
LogRateLimitIntervalSec=0
User=$USER_NAME
Environment="PATH=/bin:/usr/bin:$USER_HOME/.local/share/solana/install/active_release/bin"
ExecStart=$USER_HOME/start-validator.sh
[Install]
WantedBy=multi-user.target
EOF
sudo tee /etc/logrotate.d/agave > /dev/null << EOF
$USER_HOME/log/agave-validator.log {
su $USER_NAME $USER_NAME
daily
rotate 3
missingok
postrotate
systemctl kill -s USR1 agave-validator.service
endscript
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart logrotate
8. Start the Validator
sudo systemctl enable agave-validator.service
sudo systemctl start agave-validator.service
sudo systemctl status agave-validator.service
tail -f ~/log/agave-validator.log
9. Monitoring Commands
Check Validator Status
sudo systemctl status agave-validator.service
sudo systemctl is-active agave-validator.service
sudo systemctl is-enabled agave-validator.service
View Logs
tail -f ~/log/agave-validator.log
tail -100 ~/log/agave-validator.log
sudo journalctl -u agave-validator -f --no-pager
grep -i error ~/log/agave-validator.log | tail -20
Check Sync Status
solana slot
curl -s https://api.mainnet-beta.solana.com -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getSlot"}'
solana catchup --our-localhost
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getHealth"}'
Check Block Storage
ls -lh /mnt/data/solana/ledger/rocksdb/
du -sh /mnt/data/solana/ledger/*
SLOT=$(solana slot)
TEST=$((SLOT - 300))
echo "Testing block from slot: $TEST"
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "getBlock",
"params": [
'$TEST',
{
"encoding": "json",
"transactionDetails": "signatures",
"maxSupportedTransactionVersion": 0
}
]
}' | head -50
Monitor Resources
df -h /mnt/data/
watch -n 10 'du -sh /mnt/data/solana/ledger/*'
free -h
top -b -n 1 | head -20
ps aux | grep agave-validator
10. Complete Monitoring Script
cat > ~/monitor-solana.sh << 'EOF'
echo "═══════════════════════════════════════════════════════"
echo " SOLANA RPC NODE MONITORING DASHBOARD"
echo "═══════════════════════════════════════════════════════"
echo "Time: $(date)"
echo ""
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m'
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "1. SYNC STATUS"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
YOUR_SLOT=$(solana slot 2>/dev/null)
MAINNET_SLOT=$(curl -s https://api.mainnet-beta.solana.com -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getSlot"}' | grep -oP '(?<="result":)\d+')
DIFF=$((MAINNET_SLOT - YOUR_SLOT))
echo "Your Node Slot: $YOUR_SLOT"
echo "Mainnet Slot: $MAINNET_SLOT"
echo "Difference: $DIFF slots"
if [ $DIFF -lt 100 ]; then
echo -e "Status: ${GREEN}✅ FULLY SYNCED${NC}"
else
echo -e "Status: ${YELLOW}⚠️ SYNCING ($DIFF slots behind)${NC}"
fi
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "2. RPC HEALTH"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
HEALTH=$(curl -s http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getHealth"}' | grep -oP '(?<="result":")[^"]*')
if [ "$HEALTH" = "ok" ]; then
echo -e "RPC Status: ${GREEN}✅ $HEALTH${NC}"
else
echo -e "RPC Status: ${RED}❌ $HEALTH${NC}"
fi
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "3. BLOCK STORAGE TEST"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
if [ -d "/mnt/data/solana/ledger/rocksdb" ]; then
echo -e "RocksDB: ${GREEN}✅ EXISTS${NC}"
TEST_SLOT=$((YOUR_SLOT - 300))
BLOCK_RESULT=$(curl -s http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "getBlock",
"params": ['$TEST_SLOT', {"encoding": "json", "transactionDetails": "signatures"}]
}')
if echo "$BLOCK_RESULT" | grep -q '"result":{'; then
echo -e "Block Retrieval: ${GREEN}✅ WORKING${NC}"
else
echo -e "Block Retrieval: ${RED}❌ FAILED${NC}"
fi
else
echo -e "RocksDB: ${RED}❌ NOT FOUND${NC}"
fi
echo ""
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
echo "4. DISK USAGE"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
df -h /mnt/data/ | tail -1
echo ""
echo "Ledger Size: $(du -sh /mnt/data/solana/ledger 2>/dev/null | cut -f1)"
echo "Accounts Size: $(du -sh /mnt/data/solana/accounts 2>/dev/null | cut -f1)"
echo ""
echo "═══════════════════════════════════════════════════════"
EOF
chmod +x ~/monitor-solana.sh
~/monitor-solana.sh
11. Service Management
Start/Stop/Restart
sudo systemctl start agave-validator.service
sudo systemctl stop agave-validator.service
sudo systemctl restart agave-validator.service
sudo systemctl enable agave-validator.service
sudo systemctl disable agave-validator.service
Check Logs for Errors
sudo journalctl -u agave-validator -p err -n 50
sudo journalctl -u agave-validator -n 100 --no-pager
sudo journalctl -u agave-validator -f
12. Troubleshooting
Validator Won't Start
sudo systemctl status agave-validator.service
tail -100 ~/log/agave-validator.log
bash -x ~/start-validator.sh
No Block Storage (rocksdb missing)
ls -lh /mnt/data/solana/ledger/rocksdb/
sudo systemctl stop agave-validator.service
sudo rm -rf /mnt/data/solana/ledger/*
sudo rm -rf /mnt/data/solana/accounts/*
sudo systemctl start agave-validator.service
Validator Not Catching Up
solana catchup --our-localhost
sudo systemctl restart agave-validator.service
tail -f ~/log/agave-validator.log | grep -E "downloaded|slot|catchup"
High Memory Usage
free -h
top -p $(pgrep agave-validator)
13. Updating Agave
agave-validator --version
agave-install update
sudo systemctl restart agave-validator.service
14. Nuclear Reset (Start Fresh)
If everything is broken and you want to start over:
sudo systemctl stop agave-validator.service
sudo rm -rf /mnt/data/solana/ledger/*
sudo rm -rf /mnt/data/solana/accounts/*
rm -rf ~/log/*
sudo systemctl start agave-validator.service
tail -f ~/log/agave-validator.log
15. Testing Your RPC
Basic Tests
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getHealth"}'
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getVersion"}'
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getSlot"}'
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1, "method":"getLatestBlockhash"}'
Test Block Retrieval
SLOT=$(solana slot)
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{
"jsonrpc":"2.0",
"id":1,
"method":"getBlock",
"params":['$((SLOT-100))', {"encoding":"json","maxSupportedTransactionVersion":0}]
}'
Test Transaction History
curl http://localhost:8899 -X POST \
-H "Content-Type: application/json" \
-d '{
"jsonrpc":"2.0",
"id":1,
"method":"getSignaturesForAddress",
"params":["vines1vzrYbzLMRdu58ou5XTby4qAqVRLmqo36NKPTg", {"limit":10}]
}'
16. Expected Timeline
First-Time Setup:
Installation: 10 minutes
Snapshot download: 2-3 hours
Snapshot unpacking: 20-40 minutes
Catching up to network: 30-60 minutes
Total: 4-6 hours
After First Sync:
Ongoing:
17. Key Indicators of Success
Working Node Checklist:
Service is active and running
/mnt/data/solana/ledger/rocksdb/ directory exists
solana slot returns a slot within 100 of mainnet
getBlock queries return transaction data
getHealth returns "ok"
Logs show no continuous errors
18. Common Mistakes to Avoid
Missing the number on --limit-ledger-size → Blocks immediately pruned
Not including --enable-extended-tx-metadata-storage → No transaction history
Typo in known-validator keys → Validator won't start (exactly 44 chars!)
Running in Docker → Performance issues
Not applying system tuning → Node struggles or fails
Insufficient disk space → Node fills disk and crashes
Not setting up log rotation → Logs fill OS disk
Support Resources
Final Notes
This setup is for mainnet RPC node (non-voting)
For testnet, change entrypoints and genesis hash
For devnet, use devnet entrypoints
Minimum hardware: 256GB RAM, 2TB NVMe, fast CPU
Recommended: 512GB+ RAM, 4TB+ NVMe, 24+ cores
Good luck! The first sync takes patience, but once it's running, it's solid.