Amazon web services – How to handle real time price fetching data for crypto and tokens on server my server memory keeps getting full after a while

You have quite a few possible reasons such as

• CoinGecko API responses are stored in apiCache. If TOKENS is large, this cache can grow unchecked. You need to limit the cache size and periodically clean up old entries.

• The WebSocket ticker for ETH, BTC, and XRP is opened but never closed and each call to client.ws.ticker creates a new connection, consuming memory. So only have one connection per token and close it when done.

• The setTimeout with a callback to processTokens can create a large number of pending callbacks if the processing of tokens takes longer than expected. This can queue up many setTimeout calls, consuming memory. You need to wait until processTokens function finishes before scheduling the next call

• If any promises are rejected without being caught properly, the unhandled rejection might cause memory issues.

Here is a suggested rewrite, note I moved the main part of the URL to the start as well.

const apiROOT = `https://pro-api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=1&page=1&sparkline=false&x_cg_pro_api_key=${config.CG_KEY}`;
const apiCache = {};
const CACHE_DURATION = 5 * 60 * 1000; // Example cache duration of 5 minutes
const CryptoConversionFactor = 8 ** 10; // No need to calculate this each time
const timeoutMS = 2000; // I would raise this too
let tokenDatas = [];
let wsConnections = {};

// Function to clean up cache
const cleanCache = () => {
  const now = Date.now();
  for (const coinid in apiCache) {
    if (now - apiCache[coinid].timestamp > CACHE_DURATION) {
      delete apiCache[coinid];
    }
  }
};

// Function to manage WebSocket connections
const manageWebSocket = (token, _openprice) => {
  if (!wsConnections[token.symbol]) {
    wsConnections[token.symbol] = client.ws.ticker(token.symbol + "USDT", (ticker) => {
      const feed = formatFeedFromTicker(ticker, _openprice);
      updatePriceFeed(token, feed, callsendprice, tokenDatas);
    });
  }
};

// Function to process tokens periodically
const processTokens = async() => {
  try {
    for (const token of TOKENS) {
      const priceData = await liquidityContract.methods
        .getPrice([token.address, token.address])
        .call();
      const _openprice = parseFloat(priceData.price) / CryptoConversionFactor;

      if (["ETH", "BTC", "XRP"].includes(token.symbol)) {
        manageWebSocket(token, _openprice);
      } else {
        if (!apiCache[token.coinid] || Date.now() - apiCache[token.coinid].timestamp > CACHE_DURATION) {
          const response = await fetch(`${apiROOT}&ids=${token.coinid}`);
          if (!response.ok) continue;
          const data = await response.json();
          if (data.length === 0) continue;
          apiCache[token.coinid] = {
            data: data[0],
            timestamp: Date.now(),
          };
        }
        const feed = formatFeedFromAPI(_openprice, apiCache[token.coinid].data);
        updatePriceFeed(token, feed, callsendprice, tokenDatas);
      }
    }

    if (broadcastMessage) {
      broadcastMessage({
        tokenData: tokenDatas
      });
    }
  } catch (error) {
    console.error("Error in processTokens: ", error);
  } finally {
    cleanCache();
    // Schedule the next run only after the current one finishes
    setTimeout(processTokens, timeoutMS);
  }
};

// Start the process
processTokens();

// Transaction checking
if (calltx) {
  monitorTransactions(callbacktx);
}