So, the idea is to use goreplay to capture the internal component traffic,

it’s seem like the application was design in the way that it’s can attached directly to server itself.

problems is we don’t really have that server because we use load balancer as a service in our production.

So access to load balancer is not possible, now the plan is to create nginx proxy box. and install goreplay on it. so all traffic will be pass via this little box before going directly into load balancer service.

to setup nginx.

Dockerfile it’s goes

# Use the official Nginx image as a base
FROM nginx:stable

# Copy the custom Nginx configuration file to the container
COPY nginx.conf /etc/nginx/nginx.conf

# Expose ports 8880 and 9990
EXPOSE 8880
EXPOSE 9990

nginx.conf

User and worker configuration

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

# User and worker configuration
user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    
    sendfile        on;
    keepalive_timeout  600;

    # Server block for port 8880
    server {
        listen 8880;
        server_name localhost;

        location / {
            proxy_pass http://10.6.50.74:8080;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Do not intercept error codes, including 429 Too Many Requests
            proxy_intercept_errors off;

            # Forward all headers from upstream server to the client
            proxy_pass_header  X-*;
            proxy_pass_header  *;
        }
    }

    # Server block for port 9990
    server {
        listen 9990;
        server_name localhost;

        location / {
            proxy_pass http://10.6.50.75:9090;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Do not intercept error codes, including 429 Too Many Requests
            proxy_intercept_errors off;

            # Forward all headers from upstream server to the client
            proxy_pass_header  X-*;
            proxy_pass_header  *;
        }
    }
}

Then build and run the NGINX.

(at this step, due to our prod block internet so reaching dockerhub is not possible. that where we do a container image upload manually. ) Here

docker build -t nginx-proxy .

Yea

In case of update nginx.config files do the following instead.

#set all previous docker that might have restart=away and may one day up again. 
docker ps -a -q | xargs -I {} docker update --restart=no {}
#kill the running docker. 
docker ps 
docker kill <containerid> 
docker rm <containerid> 
#build new image.
docker build -t nginx-proxy . 

Troubleshoot ; check error message.

docker exec -it <nginx-container-id> cat /var/log/nginx/error.log

Check if upstream server is good, run command to ping from the pod.

docker exec -it <nginx-container-id> ping 10.6.50.74
docker exec -it <nginx-container-id> telnet 10.6.50.74 8080

now make sure docker run on boot

docker run -d --restart always -p 8888:8880 -p 9999:9990 nginx-proxy

now download goreplay to /sbin folder.

just use filezilla or some similar thing. 

now make goreplay run.

sudo /home/azureuser/gor --input-raw :9999 --output-file-append --output-file /media/goreplay/latest/goreplay-9999-%Y-%m-%d--%H-%M-UTC.gz
sudo /home/azureuser/gor --input-raw :8888 --output-file-append --output-file /media/goreplay/latest/goreplay-8888-%Y-%m-%d--%H-%M-UTC.gz

now if you want to capture a lot of data, mount an SMB drive. .

sudo apt update
sudo apt install cifs-utils
sudo apt install autofs

Setup credential

sudo mkdir -p "/etc/smbcredentials"
sudo nano /etc/smbcredentials/$STORAGE_ACCOUNT_NAME 
sudo chmod 600 /etc/smbcredentials/$STORAGE_ACCOUNT_NAME  

#file content for /etc/smbcredentials/stbimsuatnw 
username=$STORAGE_ACCOUNT_NAME
password=$STORAGE_ACCOUNT_KEY

Update autofs configuration file

sudo su 


sudo nano /etc/auto.fileshares 

goreplay  -fstype=cifs,credentials=/etc/smbcredentials/stbimsuatnw,serverino,nosharesock,actimeo=30,mfsymlinks ://stbimsuatnw.privatelink.file.core.windows.net/goreplay


sudo nano /etc/auto.master
/media /etc/auto.fileshares --timeout=60

To test the drive mount. try this (optional

mount -t cifs //stbimsuatnw.file.core.windows.net/goreplay/ /media/goreplay -o credentials=/etc/smbcredentials/stbimsuatnw

then restart autofs to mount the drive

sudo systemctl restart autofs
sudo systemctl enable autofs

Now make this whole thing as a service.

Create folder in share drive to support the script.

mkdir /media/goreplay/latest

failure backoff script.

sudo nano /home/azureuser/run_gor_commands.sh

#!/bin/bash

# Function to run a command with retries and backoff, logging to /var/log/goreplay.log
run_with_backoff() {
    local cmd="$1"
    local initial_delay=1
    local max_delay=32
    local delay=$initial_delay
    mkdir -p "/media/goreplay/$(date +%Y-%m-%d)"
    while true; do
        echo "$(date '+%Y-%m-%d %H:%M:%S') - Running command: $cmd" | tee -a /var/log/goreplay.log
        mkdir -p "/media/goreplay/$(date +%Y-%m-%d)"
        # Run the command and log errors if any
        eval "$cmd" 2>> /var/log/goreplay.log
        if [ $? -eq 0 ]; then
            echo "$(date '+%Y-%m-%d %H:%M:%S') - Command succeeded: $cmd" | tee -a /var/log/goreplay.log
            delay=$initial_delay
        else
            echo "$(date '+%Y-%m-%d %H:%M:%S') - Command crashed: $cmd" | tee -a /var/log/goreplay.log
            echo "$(date '+%Y-%m-%d %H:%M:%S') - Waiting $delay seconds before retrying..." | tee -a /var/log/goreplay.log
            sleep $delay
            delay=$((delay * 2))
            if [ $delay -gt $max_delay ]; then
                delay=$max_delay
            fi
        fi
    done
}

# Run both commands in the background with retries
run_with_backoff "sudo /home/azureuser/gor --input-raw :9999 --output-file-append --output-file /media/goreplay/%Y-%m-%d/goreplay-9999-%Y-%m-%d--%H-%M-UTC.txt" &
run_with_backoff "sudo /home/azureuser/gor --input-raw :8888 --output-file-append --output-file /media/goreplay/%Y-%m-%d/goreplay-8888-%Y-%m-%d--%H-%M-UTC.txt" &

wait

CHMOD above

sudo chmod +x /home/azureuser/run_gor_commands.sh

create Systemd service

sudo nano /etc/systemd/system/goreplay.service

following content for goreplay.service

[Unit]
Description=Gor Replay Service
After=network.target

[Service]
ExecStart=/home/azureuser/run_gor_commands.sh
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=goreplay
Restart=always
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target

reload start script.

sudo systemctl daemon-reload
sudo systemctl restart goreplay.service
sudo systemctl status goreplay.service

crontab to create new folder.

0 0 * * * mkdir -p "/media/goreplay/$(date -d '+1 day' +\%Y-\%m-\%d)"

Try replaying.