Testing Plan

Li Yi
SCISM
South Bank University
08 December 1999

Contents

Testing Plan
1. Components of a C/S Test
1.1 The Server
1.2 The Client
1.3 The Network

2. Prepared Tests
2.1 Basic Tests
2.2 ASP Tests
2.3 Web Database Tests

3. Configuration for the tests
3.1 Server Memory
3.2 Network Conditions
3.3 Number of Clients

4. Single Table Queries
1. 2a. 2b. 2c. 2d. 3. 4a. 4b. 4c. 4d. 5. 6a. 6aa. 6ab. 6ac. 6b. 6ba. 6bb. 6bc. 6c. 6ca. 6cb. 6d. 6da. 6db. 7. 8. 9.

5. Join Table Queries
6. Performance monitor

1. Components of a C/S Test

The tests simulate the activity of a Web server and its many client Web browsers' communication across one network.

A test involves three primary subsystems: a server, a client, and the network. During a test, a client run a virtual client program and the server responds with the result files. This section describes these components and the programs they run.

Figure 1 shows the pieces involved in a single-network test.

1.1 The Server

The server is typically an IBM-compatible PC with Internet Information Services, SQL Server 6.5, and Windows NT 4.0 Server. The server uses sets of prepared sample ASP pages and sample databases, which simulate those that a server might provide to its clients. The prepared ASP pages and databases simulate Web databases of varying lengths.

The server performance is investigated by a broad spectrum of client demands. Alternatively, the test may be repeatedly run the same prepared test, before and after hardware and software changes, provided this choice is available.

In a test, the server:

Responds to requests for connections.
Establishes and manages connections.
Receives requests for Web content.
Processes client requests and connects to a database server.
Processes database requests and sends responses.

The specification of the server is:

Pentium Pro 200, L1 cache and L2 cache, 64MB, 3GB EIDE HDD, and 10Mbps 3Com Ethernet NIC
Windows NT 4.0 Server, Internet Information Server 4.0, Active Server Pages 2.0, SQL Server 6.5, and TCP/IP support

1.2 The Client

The client consists of a computer running a client application (see the attached program file). The client application certainly can be run on more than one client computer.

The client application runs in a single, multithreaded process. Each thread in the process represents a virtual client. Each virtual client makes one connection and page request to the server at a time. This design enables each client computer to simulate more than one client.

If a client is using more than the total amount of physical memory it has available by running a large number of virtual clients, the resulting use of virtual memory reduces the speed at which it can send requests. The number of virtual clients the computer supports should be scaled back until it is using only physical memory.

The specification of client is:

Pentium 200MMX, 64MB, 3GB EIDE HDD, and 10Mbps 3Com Ethernet NIC
Windows NT 4.0 workstation, and TCP/IP support

In a test, the tester specify what level of client demand the server is subject to, including:

The number of client browsers in the test.
The pages the clients request.
The rate at which clients send requests.

The client application initiates by using an input file described in the following section. When the test is over, the application writes the test results into an output file.

The Input file

The input file is a text file, which is stored on the client computer. It specifies the number of virtual clients that will run. The request pages and the request rate are specified also. For a sample the input file, see the attached file.

The Output file

The testing results are recorded in the output file, including the time of sending request, the time of receiving request, and the database server running time. The sample is shown at the attached file.

1.3 The Network

In this plan, the network refers to the communication links between the client computers, and the server computer during a test. It is a 10Mbps Ethernet network connecting by a hub.

2. Prepared Tests

There are three types of tests:

Basic tests. These tests monitor the response of servers performing different tasks involving static web pages.
ASP tests. These tests monitor the response of the server when it is generating web pages dynamically.
Web database tests. These tests monitor the response of the server when it is connecting to a database server.

2.1 Basic Tests

Tests the response of the server to repeated requests for the same 1K file individually. This test can be used to monitor how the server uses the Internet Information Services (IIS) Object Cache and the file system cache.
Tests the response of the server to an increasing number of clients. The test begins with the minimum number of client computers requesting a single 1K file. As the test progresses, more client computers are added, each requesting the same 1K file, Until the maximum number of client computers is reached.
Tests the response of the server to a typical workload the average page size for this test is about 10k, which is commonly observed on most servers on the Internet.
Tests the response of the server to a typical very large workload. The files range in size from 256 bytes to 1MB. The total workload is approximately 200MB. If the file system cache on the server computer cannot contain all the files in this workload, the server will be occupied with reading files from disk. Increase the server memory or adjust the performance expectations accordingly.

2.2 ASP Tests

ASP applications are programs that produce dynamic web pages. The tests use a set of eight files: App.asp, Session.asp, QueryStr.asp, Form.asp, Hello.asp, Fibo.asp, Sessid.asp, and Server.asp.

Since ASP has session function, cookies are useful. If a virtual client sends a series of requests to an ASP file and cookies are turned off, then ASP has to create a new session object for each request. This causes increased overhead on the server. If cookies are turned on, ASP can reuse the session, as it does with actual browsers; when the client requests an ASP file, the server puts the session ID into a cookie and sends it to the client. The client stores the cookie and sends it with any subsequent requests to the server.

Tests the response of the server to a request for an ASP page. In this test, 100% of client requests are for files that require processing by ASP.DLL.
Tests the response of the server to a request for an ASP page. In this test, 50% of client requests are for files that require processing by ASP.DLL. The remaining 50% of client requests are taken from Basic test 2.

2.3 Web Database Tests

The test should consider the functions of cache, so that some data is being read from disc rather than from cache. For the purposes of clarity some tests assume the table in question is in the cache.

All tests to be carried out on the tables, which fit on one disc. For all tests a 30k table with a large (2 kilobytes) tuple length will be used as well as a table with a short (50 byte) tuple length. This is to check how the size of tuple effects.

Scanning a database page.
Numerical predicate check on non-key attribute
Numerical predicate check on key attribute
String predicate check
Max aggregation
Min aggregation
Returning result tuples locally
Sum aggregation
Ave aggregation
Grouping tuples
Sorting tuples
Checking index blocks
Checking non-unique index blocks
Inserting tuples into a table
Deleting tuples from a table
Updating tuples in tables
Paging
Filtering
Batch updates

3. Configuration for the tests

The tests are suggested to be repeated several different tests under different conditions. Vary the speed and number of processors (if available), the amount of physical memory, the available hard disk space and disk configuration, and network bandwidth and traffic.

3.1 Server Memory

Make sure that the system has sufficient memory for the tests. It is recommended that the Server computer should have enough physical memory to contain all static content pages involved in the test simultaneously.

3.2 Network Conditions

A network with limited bandwidth or substantial traffic can limit the performance of the server. The ideal network condition (100Mbps isolated network) is suggested and it is not a factor in the test.

IIS Bandwidth Throttler can be used to change the bandwidth available to Internet Information Services, if different network conditions are to be simulated.

3.3 Number of Clients

The number of virtual clients in the test should be varied to simulate different conditions.

4. Single Table Queries

Looking for the cost of scanning the whole table by checking a non-key attribute and returning no tuple.

SELECT *
FROM ta30
WHERE entier < -5,000,000,000
T₁ = 30k*T_scan + 30k*T_large

2a.

Looking for the cost of scanning the whole table by checking the key attribute and returning no tuple.

SELECT *
FROM ta30
WHERE cle < 0
T_2a = 30k*T_scan + 30k*T_small

2b.

Looking for the cost of scanning the whole table by checking the key attribute twice and returning no tuple.

SELECT *
FROM ta30
WHERE cle < 0 OR cle>9,999,999,999
T_2b = 30k*T_scan + 30k*T_small+ 30k*T_large

2c.

Looking for the cost of scanning the whole table by checking the key and a non-key attribute and returning no tuple.

SELECT *
FROM ta30
WHERE cle < 0 OR entier<-5,000,000,000
T_2c = 30k*T_scan + 30k*T_small+ 30k*T_large

2d.

Checking that the order of the predicates. It does not matter.

SELECT *
FROM ta30
WHERE entier<-5,000,000,000 OR cle < 0

T_2d = 30k*T_scan + 30k*T_small+ 30k*T_large

Looking for the cost of a scan. Checking two non-key predicates and returning no tuple.

SELECT *
FROM ta30
WHERE entier<-5,000,000,000 OR signe < 0

T₃ = 30k*T_scan + 30k*T_small+ 30k*T_large

T_large= T₃ - T₂ , T_scan = T₂ - T_{large ,}T_small = T₃ - T₁

4a.

Looking for the cost of returning 30000 result tuples locally.

SELECT *
FROM ta30
WHERE entier>-5,000,000,000

T_4a = 30k*T_scan + 30k*T_large+ 30k*T_return

(I doubt the success of returning so many tuples.)

4b.

Looking for the cost of returning one tuple locally and an equality check.

SELECT *
FROM ta30
WHERE entier = 888888888

T_4b = 30k*T_scan + 30k*T_small-equal+ 1*T_return

4c.

Looking for the cost of returning one tuple locally and an equality check.

SELECT *
FROM ta30
WHERE entier = 111111

T_4c = 30k*T_scan + 30k*T_small-equal+ 1*T_return

4d.

Looking for the cost of returning one result tuple locally and an inequality check.

SELECT *
FROM ta30
WHERE entier < 111111

T_4d = 30k*T_scan + 30k*T_small+ 1*T_return

Looking for the cost of checking a string

SELECT *
FROM ta30
WHERE jour = 29-10-34

T₅ = 30k*T_scan + 30k*8*T_small+ 1*T_return

6a.

Looking for the cost of max aggregation.

SELECT max(entier)
FROM ta30
WHERE cle < 0

T_6a = 30k*T_scan + 30k*T_small+ 1*T_agg+ 1*T_return

agg is a max or a min aggregation swap, which involves swapping the current value for the most recently scanned, this value is assumed to be fifty percent of all tuples as the table is created with random values.

6aa.

Looking for the cost of the max aggregation.

SELECT max(entier)
FROM ta30
WHERE cle > 0

T_6aa = 30k*T_scan + 30k*T_small+ 30k*T_scan + 30k*T_small+ 15k*T_swap+ 1*T_return

6ab.

Looking for the cost of the max aggregation.

SELECT max(entier)
FROM ta30
WHERE entier > -5000000000

T_6ab = 30k*T_scan + 30k*T_small+ 30k*T_scan + 30k*T_small+ 15k*T_swap+ 1*T_return

6ac.

Looking for the cost of the max aggregation.

SELECT max(entier)
FROM ta30
WHERE entier > Y

T_6ac = 30k*T_scan + 30k*T_small+ Y*T_scan + Y*T_small+ Z*T_swap+ 1*T_return

Where Y will be a number of tuples passing the check and Z is the number of swaps, which will be carried out.

6b.

Looking for the cost of the min aggregation.

SELECT min(entier)
FROM ta30
WHERE cle < 0

T_6b = 30k*T_scan + 30k*T_small+ 1*T_agg+ 1*T_return

6ba.

Looking for the cost of the min aggregation.

SELECT min(entier)
FROM ta30
WHERE cle > 0

T_6ba = 30k*T_scan + 30k*T_small+ 30k*T_scan + 30k*T_small+ 15k*T_swap+ 1*T_return

6bb.

Looking for the cost of the min aggregation.

SELECT min(entier)
FROM ta30
WHERE entier > -5000000000

T_6bb = 30k*T_scan + 30k*T_large+ 30k*T_scan + 30k*T_small+ 15k*T_swap+ 1*T_return

6bc.

Looking for the cost of the min aggregation.

SELECT min(entier)
FROM ta30
WHERE entier > Y

T_6bc = 30k*T_scan + 30k*T_small+ Y*T_scan + Y*T_small+ Z*T_swap+ 1*T_return

Where Y will be a number of tuples passing the check and Z is the number of swaps, which will be carried out.

6c.

Looking for the cost of the sum aggregation.

SELECT sum(entier)
FROM ta30
WHERE cle < 0

T_6c = 30k*T_scan + 30k*T_small+ 1*T_sum+ 1*T_return

6ca.

Looking for the cost of the sum aggregation.

SELECT sum(entier)
FROM ta30
WHERE cle > 0

T_6ca = 30k*T_scan + 30k*T_small+ 30k*T_sum + 1*T_return

6cb.

Looking for the cost of the sum aggregation.

SELECT sum(entier)
FROM ta30
WHERE entier > -5000000000

T_6cb = 30k*T_scan + 30k*T_large+ 30k*T_sum + 1*T_return

6d.

Looking for the cost of the avg aggregation.

SELECT avg(entier)
FROM ta30
WHERE cle < 0

T_6d = 30k*T_scan + 30k*T_small+ 1*T_avg+ 1*T_return

6da.

Looking for the cost of the avg aggregation.

SELECT avg(entier)
FROM ta30
WHERE cle > 0

T_6da = 30k*T_scan + 30k*T_small+ 30k*T_avg + 1*T_return

6db.

Looking for the cost of the avg aggregation.

SELECT sum(entier)
FROM ta30
WHERE entier > -5000000000

T_6db = 30k*T_scan + 30k*T_large+ 30k*T_avg + 1*T_return

Looking for the cost of the grouping.

SELECT max(name)
FROM ta30
WHERE cle < 5000000000

GROUP BY signe

T₇ = 30k*T_scan + 30k*T_large+ 10*3k*T_group + 10*3k*T_agg + 10*T_return

Looking for the cost of the sorting.

SELECT *
FROM ta30
WHERE cle < 5000000000

ORDER BY cle

T₈ = 30k*T_scan + 30k*T_large+ 30k*T_sort + 30k*T_return

Confirming the cost of the sorting.

SELECT *
FROM ta30
WHERE signe < 6

ORDER BY cle

T₉ = 30k*T_scan + 30k*T_large+ 15k*T_sort + 15k*T_return

7. Join Table Queries

8. Multi-user Tests

In order to generate a representative WWW workload, WebStone or TPC/W should be considered. The problem is how to customise the standard benchmark to the Web database requests.

Throughput or response time versus number of client

The following two kinds of tests should be carried out.

1. To fix the set of workload, and vary the number of clients concurrently accessing the server.

2. To fix the number of clients accessing the server and vary the workload.

9. Performance monitor