Reactjs – Appending data to the start of list reloads the entire list in react

I have an array of items which represents an image with each item containing image data& metadata such as the unique key and the image link. My goal is to display the list of images on screen and whenever I scroll down it loads the next set of data and vice versa when I scroll up. The problem is when I scroll down and append the next set of images it works fine, the existing images stay as is. But when I scroll up, the existing images in the DOM are reloaded again. If there are lots of images, this slows the application.

This is how the list of images are displayed:

function ImageList(props) 
{
const [imageItems, setItems] = useState([]);
return {imageItems && imageItems.map((newItem, index) => (
            <>
              <AppImage key={newItem.Id}></AppImage>
</>
}

How the items are being appended:

setItems(imageItems.concat(resultArrayFromAPI))

How the items are being added at the start

setItems(resultArrayFromAPI.concat(imageItems))

The 2nd way leads to reloading all the images again

I have tried to wrap the AppImage component in react.memo but it doesn’t work. I want the behavior to be same when adding items at the start