XmlHttprequest packet loss and reordering

[fiveworlds]

Hey, I am generating a stream of data and using xmlhttp to send data to php backend to be saved on a new line. Each line is numbered incrementally sometimes I have instances where 6 is after 8 in my file and sometimes the line is missing entirely or has truncated data and messes up the whole program. Note that the xmlhttp requests are only sent locally and not over the internet so I am unsure why some lines would be weird like that. Has anybody here had a problem with this before? There could be maybe 100000+ requests but even one messing up is a problem.

 

It seems to work fine in edge browser but chrome misses some requests and firefox is missing a lot of requests and I am not sure why there is zero errors being thrown. Below is an image of the problem as you can see the data is getting truncated which results in swapping numbers and class names and even one bug messes everything up.

 

 

If it helps the data I am getting from php is in csv format

 

number,classname,etc...

 

so in the case of the above image

 

number, and half of greys was missing.

Edited June 27, 2017 by fiveworlds

[Sensei]

Without seeing your source code, either JavaScript and PHP, it's impossible to tell where might be problem...

If it's huge program, truncate it to working-but-showing-problem the simplest possible version.

While using Internet, no doubt, you will get some packets prior other packets, if you're not waiting for receiving reply with confirmation message (async sending).

Because they go different routes, dynamically chosen by TCP/IP protocols and routers, which you cannot control.

 

https://www.w3schools.com/xml/xml_http.asp

As we can see, in this example, you're free to send 2nd, 3rd, and so on, packets, inside of onreadystatechange() method.

If you will make new request, you can't be sure which one will arrive first and which second.

 

 

IMHO, your PHP script should be prepared to receive misordered packets.

 

If you will make object, and then use the same variable in loop for storing yet another object (in the next cycle of loop), garbage collector might free 1st object..

 

Maybe this will help a bit:

https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests

Edited June 27, 2017 by Sensei

[fiveworlds]

So I put all the data into a single request but that messes everything up just as much.

 document.getElementById("generatorform").addEventListener('submit', function (ev) {
  ev.preventDefault();
  if (document.getElementById("generator").value % 1 === 0) {
   resetRepeats();
   var spins;
   for (spins = 0; spins < document.getElementById("generator").value; spins = spins + 1) {
    generator(Math.floor(Math.random() * 37), spins);
   }
   formData = new FormData();
   formData.append("formData", edata);
   request = new XMLHttpRequest();
   request.open("POST", "request.php");
   request.send(formData);
   request.onreadystatechange = function () {
    if (this.readyState === 4 && this.status === 200) {
     
     edata = "";
    }
   };
   fillRepeats(spins);
  }
 });

​I'm guessing it is because TCP has a max packet size so therefore it is cutting the data into sections and reassembling it out of order. It is basically a slot machine which can generate loads of spins.

Edited June 27, 2017 by fiveworlds

[Sensei]

How on Earth we're supposed to test your snippet here locally, if you're truncating so much?

Give proper WORKING example. Which will be WORKING stand alone on our machines..

 

Make folder, copy whatever is needed, test whether it's working/not-working-but-showing-problem, and archive it, and attach in reply..

 

​I'm guessing it is because TCP has a max packet size so therefore it is cutting the data into sections and reassembling it out of order.

No, that's not possible. Data available to application using TCP sockets (send()/recv()), are always in proper order, or they are completely missed after some packet lost.

IIRC, I had to make loop of recv(), because data were sent in blocks. And recv() is returning size of block, so there is needed to check whether it's -1 (SOCKET_ERROR) or not, and fill dynamic allocated buffer.

However UDP/ICMP can have misordered/missed data. UDP is used to send audio/video data, and miss of packet just means noise/trash on the screen/speaker, and nothing seriously wrong.

But TCP does not allow such errors. Otherwise whatever you would download from the net, could contain transmission errors, and application wouldn't have idea about it.

TCP is error checking/bothering protocol, while UDP/ICMP are not error checking/bothering protocols.

 

If you have synchronous XMLHttpRequest,

you must fill onreadystatechange field with function prior calling send() (and perhaps open())..

Isn't send() waiting for reply (that's nature of synchronous transmission, isn't?)

You have misordered initialization in the above snippet IMHO.

 

 

Like, I said, in 1st reply, make array of requests (just for test!), with incremented index to array,

send async msg to PHP, increment index, make new request (request[index+1] is completely different variable than request[index], so garbage collector won't free it).

Edited June 27, 2017 by Sensei

[Strange]

Have you checked that the data is actually being sent in the order you think?

The other possible/likely problem is with pointers to buffers. The sort of pattern of random bits of data being wrong/missing is typical of a buffer allocation problem. How are you allocating the buffer that the received data goes into? Are you sure the pointer you are using to read that data is pointing to the right place? Is the buffer being created in a called function and then used by the caller (in which case it will get corrupted)? Check the value of the pointer when the data is received and when you read it / print it out.

[fiveworlds]

Give proper WORKING example. Which will be WORKING stand alone on our machines..​

 

 

Okay, that is no problem.

 

The sort of pattern of random bits of data being wrong/missing is typical of a buffer allocation problem

 

 

Yeah maybe if I wrote changed the javascript to buffer the data and send in blocks of a fixed size like 100 lines or something. It shouldn't be hard to test.

 

roulette.zip

Edited June 28, 2017 by fiveworlds

[Sensei]

I put request.php on my external server,

updated the all URLs in the script.js (make one const at the beginning of script to contain URL, will be easier to move code),

ran it, pressed refresh, pressed generate send, and so on..

 

Is it how it should looks like.. ?

 

I don't see these greys, reds, text fields like you showed in the 1st post.

 

Pressing arrows should scroll entire table by one entry? It does not happen. Logging confirms PHP is called.

 

I noticed NaN in the bottom list:

"Average number of spins before 34 repeats 47.523809523809526

Shortest Number of spins before 34 repeats NaN

Longest Number of spins before 34 repeats NaN"

 

I put logging code in request.php to see when and whether they are called:

<?php
if(isset($_POST['formData']))
{
file_put_contents("status.txt",file_get_contents("status.txt")."formData\r\n");

	file_put_contents("data.txt",$_POST['formData']."\n",FILE_APPEND);
}
else if(isset($_POST['scroll']))
{

file_put_contents("status.txt",file_get_contents("status.txt")."scroll\r\n");
	$fp = fopen("data.txt","r");
	$count = 0;
	while($count < $_POST['scroll'])
	{
		fgets($fp);
		$count=$count+1;
	}
	$count = 0;
	while($count < 33){
		$count++;
		echo fgets($fp);
	}
	fclose($fp);
} else if(isset($_POST['reset'])){

file_put_contents("status.txt",file_get_contents("status.txt")."reset\r\n");
	unlink("data.txt");
}
?>

Scroll, Reset and formData commands arrive to PHP script from what I see in status.txt.

 

ps. Strange, good advice with this double-buffering while using async. +1

 

ps2. You could add code to see status.txt file from inside of JavaScript, to immediately see what has been transmitted to PHP.

 

 

 

Ok. When I entered 1 in Generate text input field, and started pressing button, there were generated column by column data.

Until 37 column it's grey,red,black or green background.

After 37 column it's flood with blue background.. ?

 

Edited June 28, 2017 by Sensei

[fiveworlds]

I don't see these greys, reds, text fields like you showed in the 1st post.​

 

 

It only needs to use php to scroll. It generates all values fine and console.log gives the correct value but when php saves the data it is messed up.

 

Pressing arrows should scroll entire table by one entry? It does not happen.

 

 

The scrolling should work if the data saved in data.txt is correct. Javascript should send php the line number and get 33 lines back which then are used to update the table. That said the php does need to be updated to handle when you are near the end of the file. I haven't written that yet because of this problem. Also even if the data saved in the data.txt file is 100% correct it will still sometimes fail to retrieve the data correctly.

 

After 37 column it's flood with blue background.. ?

 

It is supposed to be like that.

Edited June 28, 2017 by fiveworlds

[Sensei]

But how to see problem you're describing in the 1st post.. ?

What to press? In what order?

 

Why are you using data.txt file, instead of f.e. MySQL database for storing these records.. ?

Edited June 28, 2017 by Sensei

[fiveworlds]

Why are you using data.txt file, instead of f.e. MySQL database for storing these records.. ? ​

 

 

I would definitely be using MySQL once it is finished this isn't anywhere near finished. file_put_contents is one line of code that makes it easy to see exactly what data I am getting. Other changes that could be made would be that the generator function is updating the dom alot and it should only really need to update it once. Also I probably don't need to send as strings xmlhttp allows you to send typed arrays of 8 and 16 bit integers which would be less data to send.

 

 

What to press? In what order?

 

 

 

Refresh then. Generator input the number 10000 wait for it open data.txt (I am using notepad++) and you should have at least one error in the file. Low numbers may not generate the error but 10000 should.

Edited June 28, 2017 by fiveworlds

[Sensei]

IMHO, it's wrong design since the beginning of project.

 

Much easier for you would be if JavaScript would be just making XMLHttpRequest, and in onreadystatechange() just set div tag content to whatever PHP script returned.

And generate entire HTML table inside of PHP script.

Trying to generate entire table, with rows, with columns, inside of JavaScript, overbloated entire project considerably.

While in PHP it would be just printf( "<table><tr>%s</tr></table>", $row ); etc. etc.

Edited June 28, 2017 by Sensei

[fiveworlds]

IMHO, it's wrong design since the beginning of project.

 

Yeah I knew that before I posted just wanted to know if there was a solution to the xmlhttp problem I ran into. This project might be ok to put into php but I may run into other instances where I might need xmlhttp to be a bit more reliable.

 

While in PHP it would be just printf( "<table><tr>%s</tr></table>", $row ); etc. etc.

 

 

Only xmlhttp is unreliable so the less data you have to send with it the better so generating the html with javascript is ok what I should have done though is have the generator function in php, only print the 33 lines I need and use numbers for the classnames.