Thursday, February 22, 2007

Status Report for 02/21/2007

I have listed some questions and/or concerns coming from Marlon Pierce as below. I am going to upgrade and post the new document in accordance with these feedbacks soon. Here you see the brief answers.

* You need to convert this into a plan of action ("I will test idea #1 by doing x,y, and z...").
Answer :
I will extend the document with this information. I am planning to make scalability tests over Pattern Informatics application.

* Critical regions: you have three independent things to investigate here. First, how do you determine a critical region? It will vary depending on the layers, I think, so you need a strategy for that (ie two different layers could have two different critical regions).Second, assume that you have a critical region and apply the approach outlined in your report.Third, assume 2+ critical regions (corresponding to different layers). What happens to Fig 3 and Fig 4 in this case? I assume actually this is additive.So actually #2 above is the place to start, and then #3, and then back to #1. #1 is hardest, since defining a critical region automatically is a fuzzy decision.
Answer : I realized that I need to give more architectural details in my document. According to my architecture defined in section 1 and 2 in the document, it wont make any difference whether critical regions are applied to one or more than one layer. Same load balancing architecture will be applied to the layers separately and merged at the Mapping Server. We do not take the layers in Layer-1 (set of layers in raster format such as Google Maps and NASA satellite maps) into considerations for the load balancing. From our experience we know that creating layers for Layer-2 is the cumbersome. Therefore, load balancing is applied only to layers in Layer-2 (see the Figure-1 in the document). Layer-2 is created from the vector data which is encoded in XML structured GML format. Layer-2 might consist of more than one layer with their critical regions (as Marlon pointed out). Regarding to critical regions, layers in Layer-2 do not necessarily have critical regions defined. If it is defined, it is expected that each layer will have its own set of critical regions. If the critical region is not defined for a layer, request from client is not going to be partitioned (common way). I am going to give more detail about my proposed load balancing architecture in my upcoming updated document. I will show how to partition queries based on critical regions over a sample request having more than one layers in the group of Layer-2.
Regarding the definition of the critical regions, I explained it in the document.

* What is WMS-Extd?
Answer : WMS-Extd is a kind of acronym of "OGC compatible extended Web Map Server". Our extensions enable WMS to render images in much shorter time. Actually, I am not sure about using the term "extended". Since all the extensions we did to WMS are internal such as caching and load balancing, I might not need to change the name.

* Note also you could have the case in which the CR encloses the request (ie the inverse of 3b).
: Yes! You are right. I will add this case.

* How will you implement the caching? How, for example, will you store the images? How will you store the metadata so that you can see (quickly) if the image is already in hand? How will you manage cache-space? How long will images by cached?
Answer : In the document, I mentioned about caching and load balancing together from the load balancing point of view, and forgot to explain about how to implement caching separately. I will add a new section about it in my updated document. Here are the brief answers to above questions:
Web Map Servers are Web Services running on a web server as a web application. Possible clients to WMS are other WMS or browser based display applications. Our implementation of WMS is multi-threaded, so it can serve multiple clients at the same time. If I store the cached images in local file system it will cause trouble because of that all the threads share the local file system. In order to prevent this, I keep cached images as Image Class Object. Whenever I need it I convert them into images without any confusion. Each thread has its own instances of the classes.
Caching will be utilized just by the successive requests. In other words, cached image will be kept till the next request and it will be updated at every request. In order for the successive request to utilize the caching, its layer numbers and names should be the same, otherwise it will be counted as first time request and caching will not be utilized (see the doc - no cached data).
According to our caching implementation, I don't need to use any metadata to see if the image is already in hand. There will be only one object (Image Class Obj) cached in the system for one thread (for an active-session).
We do not need any cache space in the local file system. The only concern is the memory usage in case of excessive usage of the system. I will also be testing this.

Wednesday, February 07, 2007

Status Report for 02/07/2007

Dynamic load balancing for the Web Map Services through "Query partitioning for the variable sized and un-evenly distributed data".

In my previous post I had basically investigated the problems of distributed map processing and rendering of scientific data and information, and the integration framework of Map servers with the Geo-Science Grids from the interoperability, scalability and performance points of view.

Load balancing and the caching are the first methods coming to mind in order to enable scalability and good performance results. However, the implementation is not easy for the interactive geo-science applications due to the nature of the spatial data. Partitioning of feature data (represented in sets of polygons, line strings and points) is hard due to the spatial nature, varying sizes of the feature collections, and uncertainty about the query location. We also do not know the workload previously. The work is partitioned into independent work pieces, and the work pieces are of highly variable sizes. It is not possible to estimate the size of total work at a given server.

Here in my *draft* document I tried to solve these issues and give a summary of the proposed load balancing algorithm composed of query partitioning through caching and critical regions.