Background
I had a project which involved a lot of listings in tables or table like formats, it was coming up nicely up until I started to get to list 150K records all at once. I discovered my main bottle neck was the accompanying HTML because the database query actually was performing quite well. So the data itself was a couple kilobytes but the HTML alongside to render the page was about 10MB.
At the time I knew of RecyclerViews in mobile development and I sought to implement a kind of recycler view on my own for the project in JavaScript because the project was meant to be a lean one, so where we can we opted for bare bones implementation and this was one of them.
The Theory
First, I had to define three elements
An element whose height is predefined which will contain an item from the array being displayed. An element with a pre-defined maximum height which is scrollable and its total height is deterministic from the total number of elements to show The element to hold the list
Example:
The element with a predefined height of 10px (including margin and paddings) and the scrollable element’s max height of 30px we know the total height for the element is 50px
From the image above in the sample, I can forego generating the HTML footprint for the last two elements, when this is expanded to more than 1000 elements we get a great deal of server payload reduction as well as on page HTML.
The next was the logic to implement a “recycler” logic of sorts which would have the following functionality
Calculate the number of items to display Calculate a buffer height to push items down their intended scroll positionAfter that I had to do implement logic to show data from the array provided into the elements
The Implementation(Using TypeScript) The first three elements
The first three elements have the following initialization - we ensure that they have appropriate CSS stylings or we throw an error
Plain Bash C++ C# CSS Diff HTML/XML Java JavaScript Markdown PHP Python Ruby SQL class Virtua {
parentElement : HTMLElement ;
scrollableWrapper : HTMLElement ;
renderItem : HTMLElement ;
constructor (scrollableWrapperSelector: string, renderItemSelector: string ) {
const parentElement = document .querySelector (
parentElementSelector,
) as HTMLElement | null ;
if (parentElement) {
this .parentElement = parentElement;
} else {
throw new Error ("Invalid parent element" );
}
if (
this .parentElement .style .maxHeight === "" ||
this .parentElement .style .maxHeight === undefined
) {
throw new Error (
"Parent element wrapper element needs a max-height CSS styling" ,
);
}
const scrollableWrapper = document .querySelector (
scrollableWrapperSelector,
) as HTMLElement | null ;
if (scrollableWrapper) {
this .scrollableWrapper = scrollableWrapper;
} else {
throw new Error ("Invalid scrollable wrapper element" );
}
const renderItem = document .querySelector (
renderItemSelector,
) as HTMLElement | null ;
if (renderItem) {
this .renderItem = renderItem.cloneNode (true ) as HTMLElement ;
renderItem.remove ();
} else {
throw new Error ("Invalid render item element" );
}
this .data = data;
this .scrollableWrapper .style .height = `${this.data.length * this.maximumSingleItemHeight}px` ;
}
}
The recycler logic
The logic in the recycler checks the scroll position of the parent element and using the maximum element height calculates how many elements are already out of view and calculates the next set range of indexes to render.
Plain Bash C++ C# CSS Diff HTML/XML Java JavaScript Markdown PHP Python Ruby SQL attachScrollListener () {
this .parentElement .addEventListener ("scroll" , (event) => {
const element = event.currentTarget as HTMLElement | null ;
if (element) {
this .scrollTopPosition = element.scrollTop ;
this .pastElements = Math .floor (
this .scrollTopPosition / this .maximumSingleItemHeight ,
);
const renderDataIndexRange : {
startingIndex : number;
endIndex : number;
} = {
startingIndex : this .pastElements ,
endIndex : this .pastElements + this .maximumNumberOfItemsToDisplay ,
};
this .displayRecycler (renderDataIndexRange);
}
});
}
displayRecycler (renderDataIndexRange : {
startingIndex : number;
endIndex : number;
}): void {
this .scrollableWrapper .innerHTML = "" ;
for (
let index = renderDataIndexRange.startingIndex ;
index < renderDataIndexRange.endIndex ;
index++
) {
if (index === renderDataIndexRange.startingIndex ) {
this .scrollableWrapper .style .paddingTop = `${this.maximumSingleItemHeight * index}px` ;
const node = this .generateNode (this .renderItem , this .data [index]);
this .scrollableWrapper .append (node);
} else {
const node = this .generateNode (this .renderItem , this .data [index]);
this .scrollableWrapper .append (node);
}
}
}
generateNode (node : HTMLElement , data : any): Node {
const renderNode = node.cloneNode (true ) as HTMLElement ;
const boundElements = renderNode.querySelectorAll (
"[data-virtua-render]" ,
) as NodeListOf <HTMLElement >;
for (const boundElement of boundElements) {
const virtuaProps = boundElement.getAttribute ("data-virtua-render" );
if (virtuaProps) {
const properties = virtuaProps.split ("." );
const value = properties.reduce (
(accumulator: any, currentValue: string) => {
return accumulator == ""
? data[currentValue]
: accumulator[currentValue];
},
"" ,
);
boundElement.innerHTML = value;
}
}
return renderNode;
}
Putting it all together in one class:
Plain Bash C++ C# CSS Diff HTML/XML Java JavaScript Markdown PHP Python Ruby SQL class Virtua {
data : any[];
parentElement : HTMLElement ;
scrollableWrapper : HTMLElement ;
renderItem : HTMLElement ;
maximumNumberOfItemsToDisplay : number = 10 ;
maximumSingleItemHeight : number = 100 ;
scrollTopPosition : number = 0 ;
pastElements : number = 0 ;
constructor (
parentElementSelector: string,
scrollableWrapperSelector: string,
renderItemSelector: string,
data: any[],
{
maximumNumberOfItemsToDisplay,
maximumSingleItemHeight,
scrollTopPosition: scrollHeightPosition,
}: {
maximumNumberOfItemsToDisplay: number;
maximumSingleItemHeight: number;
scrollTopPosition: number;
},
) {
const parentElement = document .querySelector (
parentElementSelector,
) as HTMLElement | null ;
if (parentElement) {
this .parentElement = parentElement;
} else {
throw new Error ("Invalid parent element" );
}
if (
this .parentElement .style .maxHeight === "" ||
this .parentElement .style .maxHeight === undefined
) {
throw new Error (
"Parent element wrapper element needs a max-height CSS styling" ,
);
}
const scrollableWrapper = document .querySelector (
scrollableWrapperSelector,
) as HTMLElement | null ;
if (scrollableWrapper) {
this .scrollableWrapper = scrollableWrapper;
} else {
throw new Error ("Invalid scrollable wrapper element" );
}
const renderItem = document .querySelector (
renderItemSelector,
) as HTMLElement | null ;
if (renderItem) {
this .renderItem = renderItem.cloneNode (true ) as HTMLElement ;
renderItem.remove ();
} else {
throw new Error ("Invalid render item element" );
}
this .data = data;
this .scrollableWrapper .style .height = `${this.data.length * this.maximumSingleItemHeight}px` ;
if (maximumNumberOfItemsToDisplay) {
this .maximumNumberOfItemsToDisplay = maximumNumberOfItemsToDisplay;
}
if (maximumSingleItemHeight) {
this .maximumSingleItemHeight = maximumSingleItemHeight;
}
if (scrollHeightPosition) {
this .scrollTopPosition = scrollHeightPosition;
}
this .attachScrollListener ();
this .displayRecycler ({
startingIndex : 0 ,
endIndex : this .maximumNumberOfItemsToDisplay ,
});
}
attachScrollListener () {
this .parentElement .addEventListener ("scroll" , (event) => {
const element = event.currentTarget as HTMLElement | null ;
if (element) {
this .scrollTopPosition = element.scrollTop ;
this .pastElements = Math .floor (
this .scrollTopPosition / this .maximumSingleItemHeight ,
);
const renderDataIndexRange : {
startingIndex : number;
endIndex : number;
} = {
startingIndex : this .pastElements ,
endIndex : this .pastElements + this .maximumNumberOfItemsToDisplay ,
};
this .displayRecycler (renderDataIndexRange);
}
});
}
displayRecycler (renderDataIndexRange : {
startingIndex : number;
endIndex : number;
}): void {
this .scrollableWrapper .innerHTML = "" ;
for (
let index = renderDataIndexRange.startingIndex ;
index < renderDataIndexRange.endIndex ;
index++
) {
if (index === renderDataIndexRange.startingIndex ) {
this .scrollableWrapper .style .paddingTop = `${this.maximumSingleItemHeight * index}px` ;
const node = this .generateNode (this .renderItem , this .data [index]);
this .scrollableWrapper .append (node);
} else {
const node = this .generateNode (this .renderItem , this .data [index]);
this .scrollableWrapper .append (node);
}
}
}
generateNode (node : HTMLElement , data : any): Node {
const renderNode = node.cloneNode (true ) as HTMLElement ;
const boundElements = renderNode.querySelectorAll (
"[data-virtua-render]" ,
) as NodeListOf <HTMLElement >;
for (const boundElement of boundElements) {
const virtuaProps = boundElement.getAttribute ("data-virtua-render" );
if (virtuaProps) {
const properties = virtuaProps.split ("." );
const value = properties.reduce (
(accumulator: any, currentValue: string) => {
return accumulator == ""
? data[currentValue]
: accumulator[currentValue];
},
"" ,
);
boundElement.innerHTML = value;
}
}
return renderNode;
}
}
With the above, we have a really basic “virtualization” implementation which saves alot in the HTML payload. A simple initialization would be:
Plain Bash C++ C# CSS Diff HTML/XML Java JavaScript Markdown PHP Python Ruby SQL
<div class="parent-element" style="max-height: 300px; overflow-y: scroll;">
<div class="wrapper">
<div class="item">
<span data-virtua-render="foo.bar"></span>
</div>
</div>
</div>
<script>
const data = [];
for (let index = 0 ; index < 200 ; index++) {
data.push ({
foo : {
bar : 'foo ' + index
}
});
}
const virtua = new Virtua (
'.parent-element' ,
'.wrapper' ,
'.item' ,
data,
{
maximumNumberOfItemsToDisplay : 3 ,
maximumSingleItemHeight : 100 ,
});
</script>
Out of 200 items we’ll only show 3 at a time based on the above - saving about 98% of HTML payload.
Results & Takeaways
By implementing this lightweight virtual list:
DOM Nodes: Reduced from thousands down to only 3 active nodes at any time.Payload Reduction: Saved alot in rendering workload and page HTML overhead.Performance: Smooth scrolling regardless of dataset sizeReferences
https://youtu.be/EhG8bHmIkWo?si=6HJe61LvQ4mmIRAO