Explanation

This is largely dependent on a property of colour called luminance (one could conflate it with ‘brightness’), which is considered either a photometric or psychometric depending on what the colour space is.

RGB has no inherent property for luminance, rather it’s a byproduct of what your eyes do when it sees alternations in red green and blue channels. Luminance in an RGB colour model is held within the sum of its parts, so 255, 255, 0 despite not being white it’s geographically ‘close’ because two of three channels are emitting fully – you could say this has 2/3 luminance over pure white (255, 255, 255) which would be 1 – fully luminescent.

Text, as it has colour, also has a property of luminance. Text becomes difficult to read on a background when the luminance of the background is close to the luminance of the text – the difference perceptually between them is small because the difference in luminance (‘brightness’) would be too low to provoke your mind into perceiving them as distinct objects (text/background) rather than a cohesive and contiguous object (some square that changes colours, but recognizing the text within it would take a second look).

WebAIM has a colour contrast checker made specifically for developers to test this concept: https://webaim.org/resources/contrastchecker/. The contrast ratio being that aformentioned difference between the luminance of the background and the luminance of the text.

Therein lies the problem, how do we know what the luminance of a colour is if we use RGB and it has no luminance property for us?

The solution is to change your colour from one colour space to another which has that luminance property.

There are many options, but my favourite is YCoCg because of its simplicity. In YCoCg you have a Y channel (the luminance we want), as well as two color channels; Co (orange-blue channel), Cg (green-magenta channel)

The Wikipedia article for YCoCg has a code example which I’ll ‘lift’ for this answer.

To convert from RGB -> YCoCg

Co  = R - B;
tmp = B + Co/2;
Cg  = G - tmp;
Y   = tmp + Cg/2;

the reverse (YCoCg -> RGB) is as follows:

tmp = Y - Cg/2;
G   = Cg + tmp;
B   = tmp - Co/2;
R   = B + Co;

in both examples tmp is just a temporary variable you can throw away.

Using this, we can convert an RGB colour into its YCoCg equivalent and then simply look at if the luminance channel (Y) is greater than or less than 0.5. If a background colour has a Y component less than 0.5 it’s a dark colour and the text should be white, likewise if it’s greater than 0.5 it’s a bright colour and the text should then be black.

Other, more perceptual colour spaces will give you better results as they separate the luminance from the hue. Some examples are HSV and HSL (common, easy, ok results), CIE-LAB and CIE-LUV (common, difficult, good results), and OK-LAB and OK-LUV (uncommon, somewhat easy, better results).

Solution (Change background colour)

In Javascript, using YCoCg, you can change the background colour to a lighter or darker shade by moving your colour from RGB -> YCoCg, changing the Y component to something higher/brighter, and then moving it back from YCoCg -> RGB. Although I had to change my initial assumption from 0.5 to 0.75 for this one to work for most cases as blues are notorious in quick luminance calculations.

Here’s an example:

function RGBtoYCoCg(RGB) {
    const R = RGB[0];
    const G = RGB[1];
    const B = RGB[2];
    
    let Co  = R - B;
    let tmp = B + Co/2;
    let Cg  = G - tmp;
    let Y   = tmp + Cg/2;
    
    return [Y, Co, Cg];
}

function YCoCgtoRGB(YCoCg) {
    const Y = YCoCg[0];
    const Co = YCoCg[1];
    const Cg = YCoCg[2];
    
    let tmp = Y - Cg/2;
    let G   = Cg + tmp;
    let B   = tmp - Co/2;
    let R   = B + Co;
    
    return [R, G, B];
}

const eTest = document.getElementById("testelement");
const eR = document.getElementById("channelR");
const eG = document.getElementById("channelG");
const eB = document.getElementById("channelB");

function setBG() {
    const color = [Number(eR.value) / 255, Number(eG.value) / 255, Number(eB.value) / 255]; //Original color
    let colorYCoCg = RGBtoYCoCg(color); //Original color - now in YCoCg space
    
    if(colorYCoCg[0] < 0.75) //If the Y component is less than 0.75, dark...
    {
        colorYCoCg[0] = 0.75; //...then force it to be at least 0.75 'bright'
    }
    
    let colorNew = YCoCgtoRGB(colorYCoCg); //Transformed color - now back in RGB
    
    eTest.style["background-color"] = `rgb(${colorNew[0] * 255}, ${colorNew[1] * 255}, ${colorNew[2] * 255})`;
}

eR.addEventListener("input", setBG);
eG.addEventListener("input", setBG);
eB.addEventListener("input", setBG);
setBG();

<input id="channelR" type="range" min="0" max="255" />
<input id="channelG" type="range" min="0" max="255" />
<input id="channelB" type="range" min="0" max="255" />

<div id="testelement">Test bg</div>

Solution (Alternate text colour)

Another solution in Javascript, using YCoCg, can be achieved by alternating the text colour between white and black depending on the luminance:

function getLuminance(R, G, B) {
    Co  = R - B;
    tmp = B + Co/2;
    Cg  = G - tmp;
    Y   = tmp + Cg/2;
    return Y;
}

function textColor(bgRed, bgGreen, bgBlue) {
    const lumi = getLuminance(bgRed / 255, bgGreen / 255, bgBlue / 255);
    return lumi > 0.5 ? "black" : "white";
}

function setExampleRandom() {
  const eRandom = document.getElementById("random");
  const cRandom = [Math.random() * 255, Math.random() * 255, Math.random() * 255];
  eRandom.style["background-color"] = `rgb(${cRandom[0]}, ${cRandom[1]}, ${cRandom[2]})`;
  eRandom.style["color"] = textColor(cRandom[0], cRandom[1], cRandom[2]);
}

/* main */
{
  
  const eDark = document.getElementById("dark");
  eDark.style["background-color"] = "rgb(0, 20, 60)";
  eDark.style["color"] = textColor(0, 20, 60);
  
  const eLight = document.getElementById("light");
  eLight.style["background-color"] = "rgb(255, 200, 0)";
  eLight.style["color"] = textColor(255, 200, 0);
  
  const eRandom = document.getElementById("random");
  setExampleRandom();
  setInterval(setExampleRandom, 750);
}

div {
  user-select: none;
}

<main>
    <div id="dark">Test 1</div>
    <div id="light">Test 2</div>
    <div id="random">Test 3</div>
</main>