We use ajax in a number of places on our website. One instance sometimes acts irregularly as if the ajax call failed. It is just in one particular location–all the others work as expected.

When the failure occurs, the user gets an error message that the operation failed.

We cannot reproduce the failure. It always just works for us. Most users do not experience the issue. The few that do sometimes report that they tried it again later and it worked.

We can simulate the failure by renaming the ajax script. Then when the operation is attempted, the ajax call fails and we get the same error message. Yet we don’t have any reason to believe that the error is caused by inability to access the ajax script, although it could be. If that were the case, then the same issue should occur with our other ajax scripts.

We have come up with a workaround. If the user goes into incognito mode before attempting the operation, the problem is avoided.

The incognito mode workaround suggests that it might be related to a cookie. Accordingly, we have attempted to interfere with cookies in some way to try to recreate the problem. Nothing has succeeded.

To assist with troubleshooting, we have added additional diagnostic information to the error message, but so far nothing has come out of this other than information similar to "Error". Here is some code that shows how we are trying to get more information:

error: function (jqXHR, textStatus, errorThrown)
{
    // alert('read-unread error');
    ajaxError('Read-Unread', textStatus, errorThrown);  // TODO comment out to suppress the error reporting
},

For completeness, I have included the full javascript function below.

To set up a simple test environment to demonstrate the issue will not work very well, as it would simplify so much from the real environment as to destroy its test value. We think the only reasonable test would be to simulate a use case as the users are doing. Unfortunately that is complex, as it involves by creating a profile on our system, attempting to use the system to communicate with other members, having some response communication from the other members and then try to reply to such a communication. That re-creates the the use case, and the place where the failure occurs. Again, unfortunately, we suspect that it would still never trigger the failure for us–the toy system would also "just work".

It would be one thing it it happened to every user every time, but it does not. Only some users some of the time, and only for one ajax usage out of many on our site. Once it starts happening, at least for a while it is repeatable for that user.

My questions:

1. How could we create a stripped down version of the environment so we could make it worthwhile for the stackoverflow community to take a look at it, short of simulating use of our website?

2. Does the "incognito mode" workaround provide any insight to understand what is going wrong?

3. Apart from diagnosing code that in most cases works correctly, it would be helpful to get insight on other ajax failure modes. As a made up example, such as "certain systems are too slow and the ajax times out".

[The javascript function. Note: for some reason I cannot get the first few lines of the function into the code markup.]

function toggleReadUnread(usertypeid, opp_user_label_short, principalid, repid, contactuid, value) {
    var new_response = (value) ? 'read' : 'unread';
    var prefix = '#' + contactuid;

    $('#ur-read-unread-text').html(new_response);
    $('#read-unread-dialog').dialog
    ({
        show: "fade",
        title: "Change to " + new_response + "?",
        modal: true,
        resizeable: false,
        width: getDialogWidth(400, .9),
        maxWidth: 400,
        fluid: true,
        buttons:
        {
            Ok: function ()
            {
                $(this).dialog('close');
                $.ajax(
                {
                    // async: false 
                    type: 'POST',
                    url: '/ajax/set-var.php',
                    // url: '/ajax/tr-toggle-status.php',
                    data:
                    {
                        'which_function': 'read-unread',
                        'principalid': principalid,
                        'repid': repid,
                        'usertypeid': usertypeid,
                        'value': value
                    },
                    error: function (jqXHR, textStatus, errorThrown)
                    {
                        // alert(jqXHR.responseText);
                        ajaxError('Read-Unread', textStatus, errorThrown);
                    },
                    success: function (data, textStatus, jqXHR)
                    {
                        if (data)
                        {
                            var connector = '', '';
                            if (value)
                            {
                                $(prefix + '-unread').remove();
                                $('<i id="' + contactuid + '-read" class="fa fa-envelope-open tooltip" onclick="toggleReadUnread('' + usertypeid + connector + opp_user_label_short + connector + principalid + connector + repid + connector + contactuid + ''' + ', false);"><span id="' + contactuid + '-read-unread-tt" class="tooltiptext">You have no unread messages from this ' + opp_user_label_short + '</span>').insertAfter(prefix + '-read-unread-placeholder');
                            }
                            else
                            {
                                $(prefix + '-read').remove();
                                $('<i id="' + contactuid + '-unread" class="fa fa-envelope tooltip" onclick="toggleReadUnread('' + usertypeid + connector + opp_user_label_short + connector + principalid + connector + repid + connector + contactuid + ''' + ', true);"><span id="' + contactuid + '-read-unread-tt" class="tooltiptext">You have a new message from this ' + opp_user_label_short + '</span>').insertAfter(prefix + '-read-unread-placeholder');
                                $('.tc-' + contactuid).hide();
                            }
                            jConfirm('Success', 'Messages marked "' + new_response + '"');
                        }
                    }
                })
            },
            Cancel: function () {
                $(this).dialog('close');
            }
        }
    })
 }

EDIT 1:

1. We are still having this issue with users. We still cannot reproduce it internally.

2. Going incognito prevents the problem from occurring. While this should be a great clue, it has not helped us so far. It seems it points to session and caching issues. We have ruled out stale cache issues.

3. It is intermittent, and only affects a small number of users. And later even that user will no longer have the problem. Again seems like cache-related.

4. We added diagnostic info and sometimes is says "Forbidden" in the ajax errorThrown. Checking on that implies it has to do with cross domain operations. There is no cross domain activity going on; all requests are to our own server.

5. We have asked some users about their extensions. In one case, they had no extensions installed; in another case, they only had the Grammarly extension for Chrome. We tested with that extension and still could not reproduce the issue.

6. When the error occurs, a dialog is displayed. After clicking "OK" on that dialog, the rest of the operation should just proceed as there is nothing that dialog blocks. However, users report they cannot complete the operation at that point when not in incognito mode.

EDIT 2:

I started looking into the https-access.log. While most accesses to the ajax file return 200, there are cases where 302 is returned. The implication is that "file is moved". But the file has not moved and shortly thereafter normal access resumes.

Here is an extract from the access log, showing a series of 200 responses, with some 302s interspersed, and then resuming 200.

108.221.39.97 - - [13/Jan/2023:10:07:52 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:08:34 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:09:56 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
73.176.158.231 - - [13/Jan/2023:10:10:21 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:07 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
184.178.239.162 - - [13/Jan/2023:10:15:18 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?reply=tr-28560-80354" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.76"
74.73.226.238 - - [13/Jan/2023:10:15:20 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
24.144.188.195 - - [13/Jan/2023:10:15:22 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:28 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:34 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:37 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:15:49 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
72.38.12.208 - - [13/Jan/2023:10:15:54 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?reply=tr-43850-7444" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.76"
74.73.226.238 - - [13/Jan/2023:10:16:07 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
74.73.226.238 - - [13/Jan/2023:10:16:19 -0600] "POST /ajax/set-var.php HTTP/2.0" 302 - "https://www.rephunter.net/track-relationships.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:17:23 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
108.221.39.97 - - [13/Jan/2023:10:17:28 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
72.38.12.208 - - [13/Jan/2023:10:17:49 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?reply=tr-43850-7444" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.76"
108.221.39.97 - - [13/Jan/2023:10:17:58 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
24.45.242.118 - - [13/Jan/2023:10:20:41 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
24.45.242.118 - - [13/Jan/2023:10:21:10 -0600] "POST /ajax/set-var.php HTTP/2.0" 200 21 "https://www.rephunter.net/track-relationships.php?filter=clear" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"

EDIT 3:

We have a candidate solution inspired by the comment from @ibrahim to the answer from Lajos Arpad below. In short, adding retry functionality to the ajax call as shown at https://forum.framework7.io/t/app-request-what-is-the-best-way-to-retry-an-ajax-request-using-app-request/4759, which appears to be based on What's the best way to retry an AJAX request on failure using jQuery?.

After we confirm that this fix works, which will take some time of "no reported errors", I would mark it as the "answer", should @ibramim post his comment about retries as an answer.