Milestone Six Quality Assurance Report

Evolving Status Report (See JUNit Test Plan?)

Notes:

• See BugZilla
• We use enhancements to track work items; they are not bugs, they are reminders of work to do.
• Some of our low priority items are not to be fixed during the scope of our project, they are reminders for future developers.

Date: May 9th, 2005.

Progress Tests

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	2	2
2	Properties tests	9	5	5
3	Behaviour tests	18	5	5
4	Contextual Menu tests	13	0	0
5	Stub action tests	6	0	0
6	General non-editor tests	11	0	0
	Total	71	12	12

Command tests

• As of May 9th, 2005, we have created tests for 15/21 of our commands.
• All of our command tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	3	3	0
High	11	8	3
Low	15	9	6
Enhancements	30	6	24

---

Date: May 19th, 2005.

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	2	2
2	Properties tests	9	5	5
3	Behaviour tests	18	5	5
4	Contextual Menu tests	13	0	0
5	Stub action tests	6	0	0
6	General non-editor tests	11	0	0
	Total	71	12	12

Command tests

• As of May 19th, 2005, we have created tests for 14/18 of our commands.
• All of our command tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	4	4	0
High	25	22	3
Low	30	14	16
Enhancements	30	7	23

Requirements

In scope requirements (from Must Do Report)

Status	Count
Approved	9
Started	13
Implemented	14
Completed	7
Total	43

Out of scope requirements:

Status	Count
Approved	58
Started	1
Implemented	8
Completed	0
Total	67

---

Date: May 25th, 2005.

(removed one test)

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	14	9
2	Properties tests	9	9	7
3	Behaviour tests	18	6	6
4	Contextual Menu tests	13	0	0
5	Stub action tests	6	0	0
6	General non-editor tests	10	2	2
	Total	70	31	24

Command tests

• As of May 25th, 2005, we have created tests for 21/22 of our commands.
• All of our command tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	4	4	0
High	31	25	6
Low	32	21	11
Enhancements	39	8	31

Requirements

In scope requirements (from Must Do Report)

Status	Count
Approved	7
Started	14
Implemented	15
Completed	7
Total	43

Out of scope requirements:

Status	Count
Approved	58
Started	1
Implemented	8
Completed	0
Total	67

---

Date: May 31st, 2005.

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	14	12
2	Properties tests	9	8	7
3	Behaviour tests	18	7	7
4	Contextual Menu tests	13	0	0
5	Stub action tests	6	0	0
6	General non-editor tests	10	2	2
	Total	70	31	28

Command tests

• As of May 31st, 2005, we have created tests for 29/35 of our commands.
• 28 or our 29 tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	4	4	0
High	32	29	3
Low	37	21	7
Enhancements	43	24	19

Requirements

In scope requirements (from Must Do Report)

Status	Count
Approved	3
Started	8
Implemented	22
Completed	9
Total	42

Out of scope requirements:

Status	Count
Approved	58
Started	1
Implemented	8
Completed	0
Total	67

---

Date: June 1st, 2005.

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	14	12
2	Properties tests	13	13	8
3	Behaviour tests	15	14	14
4	Contextual Menu tests	12	11	8
5	Stub action tests	6	0	0
6	General non-editor tests	10	5	5
	Total	70	57	47

Command tests

• As of June 1st, 2005, we have created tests for 29/35 of our commands.
• 28 or our 29 tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	4	4	0
High	32	29	3
Low	37	21	7
Enhancements	43	24	19

Requirements

In scope requirements (from Must Do Report)

Status	Count
Approved	3
Started	8
Implemented	8
Completed	23
Total	42

Out of scope requirements:

Status	Count
Approved	58
Started	1
Implemented	8
Completed	0
Total	67

---

Date: June 14th, 2005.

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	14	12
2	Properties tests	13	13	13
3	Behaviour tests	15	14	14
4	Contextual Menu tests	12	11	9
5	Stub action tests	6	0	0
6	General non-editor tests	10	5	5
	Total	70	57	53

Command tests

• As of June 1st, 2005, we have created tests for 32/46 of our commands.
• 32 of our 32 tests pass. We do not know how many of these tests we will have when we are done.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	5	5	0
High	45	43	2
Low	48	39	9
Enhancements	47	29	18

Requirements

In scope requirements (from Must Do Report)

Status	Count
Approved	3
Started	5
Implemented	8
Completed	26
Total	42

Out of scope requirements:

Status	Count
Approved	50
Started	3
Implemented	13
Completed	0
Total	66

---

Evolution not tracked here. Could be inferred, but lack of time.

---

Date: July 14th, 2005.

Test Group Number	Test Group Name / Description	# of tests written	# of tests that can run	# of tests that pass
1	Palette tests	14	14	14
2	Properties tests	13	13	13
3	Behaviour tests	15	15	15
4	Contextual Menu tests	12	12	12
5	Stub action tests	6	1	1
6	General non-editor tests	10	8	8
	Total	70	63	63

Command tests

• As of July 14th, 2005, 36 command tests have been created. There should be 42 tests at this point.
• Some commands are used by others internally (package visibility) and don't need to be tested.
• Some other commands are marked in BugZilla as must refactor/clean and will be tested once refactored/cleaned.
• All of our programmed command tests pass.

Bug reports

Level	Bugs Found	Bugs Fixed	Bugs Remaining
Emergency	5	5	0
High	47	46	1
Low	55	46	9
Enhancements	68	40	27

• Of the remaining bugs, only five are in scope. Rest are for future work. They should be completed by the milestone date.

Requirements

In scope requirements (from Must Do Report)

Status	Count
Approved	0
Started	1
Implemented	0
Completed	42
Total	43

• The only remaining requirement is Req Elem Stub Actions's more esoteric behaviour. Requirement was too general anyways

Out of scope requirements:

Status	Count
Approved	49
Started	1
Implemented	0
Completed	15
Total	65

QA Postportem

• Quality assurance was our Achilles heel.

• We rarely implemented any tests before actually doing the coding
• Our evolving status reports shows snapshots of results right after we implement a bunch of tests.
• Our testing framework is, in my opinion, very impressive. See Command Line PDEJunit?.
  • However, we underused it.
• We found lots of bugs using our tests, but we could have found alot more had we given more effort to testing. By leaving our tests for the end, we ended up testing everything manually and then adding a few spot checks to make sure it works. This is not very useful.
• Our command tests are really useful, our progress tests, alot less. For this reason, we've dropped some of them. The hard part about progress tests is that we sometimes ended up trying to test the framework instead of our code. Hooking up to the appropriate location is the hardest part.
• Our code coverage must not be very good; don't know it exactly because Clover crashes when compiling our project.
• I recommend that further developers upgrade the testing framework to make them more useful.
• We were planning on doing code reviews weekly; we ended up doing only one.
  • Instead of having all team members participate, one member went through all the files to cleanup the code and document.
  • We did not manage to finish on time, the views package has not been cleaned.
  • We found tons of refactoring issues that are apparent once the coding is done, that we could have caught much earlier. They are now enhancement bugs in BugZilla but they could have already been fixed.

• I'm not one to make excuses, but there are two reasons for our unimpressive results concerning testing
  • Testing an Eclipse plugin is hard. Running the tests in cruisecontrol took us 20+hours. Even now, our framework (took 15+ hours of tweaking for setUp() and tearDown()) seems to have issues with how elements are initialized. We try to access things that are usually available, but fail miserably.
  • Our Quality Assurance Manager? did absolutely nothing in the last four months. Beforehand, he performed little or no QA tasks, the other team members had to do them for him. His efforts would have been crucial in ensuring a quality product.

-- Jason Kealey - 14 Jul 2005