Thursday, March 22, 2007

Status Report for 03/21/2007

Here is the link to the document explaining my research issues :
Note: All the updates I had mentioned in my previous post are done.

ABSTRACT: We have been working on creating Geo-Science grids and integrating them with Web based map and data rendering services such as WMS and Sci-Plotting. Integrating them with Geo-Science grids revealed that data rendering takes a lot of time due to querying, transforming, rendering and displaying spatial data. One can overcome these problems by caching and load balancing. However, current load balancing approaches are not suitable for rendering of variable sized and un-evenly distributed spatial data. We therefore propose a framework of "load balancing with caching" overcoming performance and scalability problems in rendering of distributed spatial data in order to make our map and plotting services feasible to use in Geo-Science grids.

CONTENT:

Thursday, March 08, 2007

Status Report for 03/07/2007

Here is the draft document explaining "dynamic load balancing with caching for spatial data rendering".

The document includes the updates based on the feedbacks listed in my previous post.

http://complexity.ucs.indiana.edu/~asayar/proposal/loadBalancing1.pdf

Abstract of the document:
We have been working on creating Geo-Science grids, integrating them with Web based map and data rendering services such as WMS and Sci-Plotting services. Integrating these services with Geo-Science grids applications revealed that data rendering takes a lot of time due to querying, transforming, rendering and displaying spatial data. One can overcome these problems by caching and load balancing. However, current load balancing approaches are not suitable for rendering of variable sized and un-evenly distributed spatial data. We therefore propose a framework of load balancing with caching to overcome performance and scalability problems in rendering of distributed spatial data.

I will be working on it more (especially for Section 3).

Thursday, February 22, 2007

Status Report for 02/21/2007

I have listed some questions and/or concerns coming from Marlon Pierce as below. I am going to upgrade and post the new document in accordance with these feedbacks soon. Here you see the brief answers.

* You need to convert this into a plan of action ("I will test idea #1 by doing x,y, and z...").
Answer :
I will extend the document with this information. I am planning to make scalability tests over Pattern Informatics application.

* Critical regions: you have three independent things to investigate here. First, how do you determine a critical region? It will vary depending on the layers, I think, so you need a strategy for that (ie two different layers could have two different critical regions).Second, assume that you have a critical region and apply the approach outlined in your report.Third, assume 2+ critical regions (corresponding to different layers). What happens to Fig 3 and Fig 4 in this case? I assume actually this is additive.So actually #2 above is the place to start, and then #3, and then back to #1. #1 is hardest, since defining a critical region automatically is a fuzzy decision.
Answer : I realized that I need to give more architectural details in my document. According to my architecture defined in section 1 and 2 in the document, it wont make any difference whether critical regions are applied to one or more than one layer. Same load balancing architecture will be applied to the layers separately and merged at the Mapping Server. We do not take the layers in Layer-1 (set of layers in raster format such as Google Maps and NASA satellite maps) into considerations for the load balancing. From our experience we know that creating layers for Layer-2 is the cumbersome. Therefore, load balancing is applied only to layers in Layer-2 (see the Figure-1 in the document). Layer-2 is created from the vector data which is encoded in XML structured GML format. Layer-2 might consist of more than one layer with their critical regions (as Marlon pointed out). Regarding to critical regions, layers in Layer-2 do not necessarily have critical regions defined. If it is defined, it is expected that each layer will have its own set of critical regions. If the critical region is not defined for a layer, request from client is not going to be partitioned (common way). I am going to give more detail about my proposed load balancing architecture in my upcoming updated document. I will show how to partition queries based on critical regions over a sample request having more than one layers in the group of Layer-2.
Regarding the definition of the critical regions, I explained it in the document.

* What is WMS-Extd?
Answer : WMS-Extd is a kind of acronym of "OGC compatible extended Web Map Server". Our extensions enable WMS to render images in much shorter time. Actually, I am not sure about using the term "extended". Since all the extensions we did to WMS are internal such as caching and load balancing, I might not need to change the name.

* Note also you could have the case in which the CR encloses the request (ie the inverse of 3b).
Answer
: Yes! You are right. I will add this case.

* How will you implement the caching? How, for example, will you store the images? How will you store the metadata so that you can see (quickly) if the image is already in hand? How will you manage cache-space? How long will images by cached?
Answer : In the document, I mentioned about caching and load balancing together from the load balancing point of view, and forgot to explain about how to implement caching separately. I will add a new section about it in my updated document. Here are the brief answers to above questions:
Web Map Servers are Web Services running on a web server as a web application. Possible clients to WMS are other WMS or browser based display applications. Our implementation of WMS is multi-threaded, so it can serve multiple clients at the same time. If I store the cached images in local file system it will cause trouble because of that all the threads share the local file system. In order to prevent this, I keep cached images as Image Class Object. Whenever I need it I convert them into images without any confusion. Each thread has its own instances of the classes.
Caching will be utilized just by the successive requests. In other words, cached image will be kept till the next request and it will be updated at every request. In order for the successive request to utilize the caching, its layer numbers and names should be the same, otherwise it will be counted as first time request and caching will not be utilized (see the doc - no cached data).
According to our caching implementation, I don't need to use any metadata to see if the image is already in hand. There will be only one object (Image Class Obj) cached in the system for one thread (for an active-session).
We do not need any cache space in the local file system. The only concern is the memory usage in case of excessive usage of the system. I will also be testing this.

Wednesday, February 07, 2007

Status Report for 02/07/2007

Dynamic load balancing for the Web Map Services through "Query partitioning for the variable sized and un-evenly distributed data".


In my previous post I had basically investigated the problems of distributed map processing and rendering of scientific data and information, and the integration framework of Map servers with the Geo-Science Grids from the interoperability, scalability and performance points of view.

Load balancing and the caching are the first methods coming to mind in order to enable scalability and good performance results. However, the implementation is not easy for the interactive geo-science applications due to the nature of the spatial data. Partitioning of feature data (represented in sets of polygons, line strings and points) is hard due to the spatial nature, varying sizes of the feature collections, and uncertainty about the query location. We also do not know the workload previously. The work is partitioned into independent work pieces, and the work pieces are of highly variable sizes. It is not possible to estimate the size of total work at a given server.

Here in my *draft* document I tried to solve these issues and give a summary of the proposed load balancing algorithm composed of query partitioning through caching and critical regions.

http://complexity.ucs.indiana.edu/~asayar/proposal/loadBalancing.pdf

Wednesday, January 24, 2007

Status Report for 01/24/2007

Here is the new *draft* proposed "Test and Experiments" document.

http://complexity.ucs.indiana.edu/~asayar/proposal/testCases1.pdf

Changes I have made since my last post dated as 12/13/06:
- I have added concrete test cases on proposed integration framework and
- Added definitions of Geo-Science Applications we have been working on.
- I have made background information more condensed and
- Given the enough detailes about how to make the usability and scalability tests

Saturday, January 13, 2007

Status Report for 12/13/2006

After getting Marlon's comments and feedbacks to my posting dated 11/29/2006, I created 5-pages of Tests and experiments document.

Morlon's brief comments on my previous posting:
-- It needs work.
-- He thinks I need to address concrete details more than general explanations such as using Web Services etc.

-- He thinks the best way to do this is write this in reverse: start at the end (the "Tests and Experiments") and work back to the beginning. This way, he thinks I won't introduce unnecessary details (like the fact that GIS is a usually a desktop application or it uses GET/POST) in the research problems section. Also, doing it this way I will not introduce side issues (like platform neutrality) that are not really researchable, things like interoperability (which is an engineering detail and also not the subject of my research), or asynchronous communication, all of which I don't address.

-- I was all aggreed on these comments.
-- In the light of these valuable suggestions I wrote below document titled as "Scalability Tests for Map Services and “Geo-Science Grids” Integration Architecture"

http://complexity.ucs.indiana.edu/~asayar/proposal/testCases1_old.doc

Monday, December 04, 2006

Status Report for 11/29/2006

In the light of our last discussions and your comments,
here is the new 2-page of *draft* abstract

http://complexity.ucs.indiana.edu/~asayar/proposal/newAbstract1.doc

I am going to put the capability based data+service integration issues a side and I will keep this problem in my mind as my long term research topic.

My current research focus is based on what I have done so far which is summarized in my previous post.

In summary, my research consists of the distributed map processing and rendering of the scientific data and information from the interoperability, scalability and performance points of view

Thanks for your comments in advance.

Friday, November 24, 2006

Summary of what I have been doing

Here in this snapshot you see the summary of my work done so far.

I am going to upload the complete presentation when I am done.

I might also enhance the outline with the definitions, motivations, research issues, and the "title" which is the most important one.





Monday, November 20, 2006

Introducing Capability Concept into Service+Data Integration

Here are the expected presentation slides. I gave the outline in my previous post.

In these slides (link is given below), I summary my project about introducing capability concept into the data+service integration problems.

Capability concept is first introduced by OGC (Open GIS Consortium) in GIS (Geographic Information Systems) domain. Their definition of capability is specific to geographic data (which is called feature) and provides ad-hoc solutions to "data access" and "service-chaining" issues in GIS domain.

The term "capability" is also used in Grid community as a tool enabling fine-grained authorization for general Web/Grid services in compliant with Web Services Specs.

http://complexity.ucs.indiana.edu/~asayar/proposal/capabilityConcept2.ppt

Thursday, November 16, 2006

Status Report for 11/16/2006

I have been preparing slides titled "Introducing Capability Concept for the Data+Service Integration".

For the concept of the slides please see the "outline" listed below.
I have finished almost everthing except for the most challenging part. This part is titled as "Research Issues". I will upload the slides as soon as I am done with this part.















Please let me know your comments and questions.

Monday, November 06, 2006

Comparing ASIS with the current data integration approaches

Here is the slides about comparing our data-integration approach with 3 other well-known approaches. These are SRB (Storage Resource Broker), Ogsa-DAI (Ogsa-DAI) and Sompel's Digital Libraries. We call our approach ASIS (Application Specific Information Systems) and it is based on OGC's capability-based service chaining.

http://complexity.ucs.indiana.edu/~asayar/proposal/dataIntegrationApproaches2.ppt

Distributed scientific and engineering applications require support for integrating data from multiple data sources through distributed query and transactions facilities. Data resides in multiple forms in numerous heterogeneous repositories and served by various kinds of data servers. In my presentation I mentioned about the data integration approaches and comparing our capability-based Application-Integration Framework (ASIS) with the other approaches.

Sunday, November 05, 2006

IVOA's Approach to Data Integration

Please see the slides presenting IVOA's approach to "Data Integration" issues. http://complexity.ucs.indiana.edu/~asayar/proposal/IVOA_DataIntegIssues.ppt

The goal of the IVOA (International Virtual Observatory Alliance) is the development of architectural decisions and standards in the astronomy domain

IVOA is not directly addressing the data integration issues, but they plan to define VOStore and VOSpace based on SRB and/or NGAS architectures. IVOA has a prototype application for the data integration, OpenSkyQuery. OpenSkyQuery is VO prototype application that marries Web Services technology with emerging VO standards to enable dynamic cross-matching queries between different VO-enabled archives. In my presentation, I tried to explain basically OpenSkyQuery and related IVOA standards and specifications.

Tuesday, October 31, 2006

Status Report for 10/31/2006

My current and future work outlined in one ppt slide :
http://complexity.ucs.indiana.edu/~asayar/proposal/capabilityConcept.ppt

Our implementation of data integration is based on geographic data model (GML) and online services (WMS and WFS) whose standard specifications are defined by OGC.

The OGC specifications are mostly domain specific and fine-grained. The specifications are dynamic and keep changing based on the communities needs and new technologies. Therefore, even in the same domain and same science community, it is very hard to create a generic capability. In order to get a rough idea about the OGC-WMS capability and how deeply it is GIS specific, please see this link (http://schemas.opengis.net/wms/1.3.0)

For example, in order to be able to apply OGC like capability based data+service integration architecture to Chemistry, a new capability schema specific to Chemistry data model (CML) should be defined by a standard body in the Chemistry community. If the community does not have the capability definitions (which is normal), and If we want OGC like capability based data+Service integration we need to define a generic (or in Chemistry domain) capability.

In order to get rid of this burden of creating different capabilities and defining different services for different domains we try to create a generic capability and architectural framework to integrate data+service in general science domains. So, these issues might be our future research.