Evolving Status Report (See JUNit Test Plan?)
Notes:
- See BugZilla
- We use enhancements to track work items; they are not bugs, they are reminders of work to do.
- Some of our low priority items are not to be fixed during the scope of our project, they are reminders for future developers.
Date: May 9th, 2005.
Progress Tests
Command tests
- As of May 9th, 2005, we have created tests for 15/21 of our commands.
- All of our command tests pass. We do not know how many of these tests we will have when we are done.
Bug reports
Date: May 19th, 2005.
Command tests
- As of May 19th, 2005, we have created tests for 14/18 of our commands.
- All of our command tests pass. We do not know how many of these tests we will have when we are done.
Bug reports
Requirements
In scope requirements (from
Must Do Report)
Status |
Count |
Approved |
9 |
Started |
13 |
Implemented |
14 |
Completed |
7 |
Total |
43 |
Out of scope requirements:
Status |
Count |
Approved |
58 |
Started |
1 |
Implemented |
8 |
Completed |
0 |
Total |
67 |
Date: May 25th, 2005.
(removed one test)
Command tests
- As of May 25th, 2005, we have created tests for 21/22 of our commands.
- All of our command tests pass. We do not know how many of these tests we will have when we are done.
Bug reports
Requirements
In scope requirements (from
Must Do Report)
Status |
Count |
Approved |
7 |
Started |
14 |
Implemented |
15 |
Completed |
7 |
Total |
43 |
Out of scope requirements:
Status |
Count |
Approved |
58 |
Started |
1 |
Implemented |
8 |
Completed |
0 |
Total |
67 |
Date: May 31st, 2005.
Command tests
- As of May 31st, 2005, we have created tests for 29/35 of our commands.
- 28 or our 29 tests pass. We do not know how many of these tests we will have when we are done.
Bug reports
Requirements
In scope requirements (from
Must Do Report)
Status |
Count |
Approved |
3 |
Started |
8 |
Implemented |
22 |
Completed |
9 |
Total |
42 |
Out of scope requirements:
Status |
Count |
Approved |
58 |
Started |
1 |
Implemented |
8 |
Completed |
0 |
Total |
67 |
Date: June 1st, 2005.
Command tests
- As of June 1st, 2005, we have created tests for 29/35 of our commands.
- 28 or our 29 tests pass. We do not know how many of these tests we will have when we are done.
Bug reports
Requirements
In scope requirements (from
Must Do Report)
Status |
Count |
Approved |
3 |
Started |
8 |
Implemented |
8 |
Completed |
23 |
Total |
42 |
Out of scope requirements:
Status |
Count |
Approved |
58 |
Started |
1 |
Implemented |
8 |
Completed |
0 |
Total |
67 |
Date: June 14th, 2005.
Command tests
- As of June 1st, 2005, we have created tests for 32/46 of our commands.
- 32 of our 32 tests pass. We do not know how many of these tests we will have when we are done.
Bug reports
Requirements
In scope requirements (from
Must Do Report)
Status |
Count |
Approved |
3 |
Started |
5 |
Implemented |
8 |
Completed |
26 |
Total |
42 |
Out of scope requirements:
Status |
Count |
Approved |
50 |
Started |
3 |
Implemented |
13 |
Completed |
0 |
Total |
66 |
Evolution not tracked here. Could be inferred, but lack of time.
Date: July 14th, 2005.
Command tests
- As of July 14th, 2005, 36 command tests have been created. There should be 42 tests at this point.
- Some commands are used by others internally (package visibility) and don't need to be tested.
- Some other commands are marked in BugZilla as must refactor/clean and will be tested once refactored/cleaned.
- All of our programmed command tests pass.
Bug reports
- Of the remaining bugs, only five are in scope. Rest are for future work. They should be completed by the milestone date.
Requirements
In scope requirements (from
Must Do Report)
Status |
Count |
Approved |
0 |
Started |
1 |
Implemented |
0 |
Completed |
42 |
Total |
43 |
- The only remaining requirement is Req Elem Stub Actions's more esoteric behaviour. Requirement was too general anyways
Out of scope requirements:
Status |
Count |
Approved |
49 |
Started |
1 |
Implemented |
0 |
Completed |
15 |
Total |
65 |
QA Postportem
- Quality assurance was our Achilles heel.
- We rarely implemented any tests before actually doing the coding
- Our evolving status reports shows snapshots of results right after we implement a bunch of tests.
- Our testing framework is, in my opinion, very impressive. See Command Line PDEJunit?.
- However, we underused it.
- We found lots of bugs using our tests, but we could have found alot more had we given more effort to testing. By leaving our tests for the end, we ended up testing everything manually and then adding a few spot checks to make sure it works. This is not very useful.
- Our command tests are really useful, our progress tests, alot less. For this reason, we've dropped some of them. The hard part about progress tests is that we sometimes ended up trying to test the framework instead of our code. Hooking up to the appropriate location is the hardest part.
- Our code coverage must not be very good; don't know it exactly because Clover crashes when compiling our project.
- I recommend that further developers upgrade the testing framework to make them more useful.
- We were planning on doing code reviews weekly; we ended up doing only one.
- Instead of having all team members participate, one member went through all the files to cleanup the code and document.
- We did not manage to finish on time, the views package has not been cleaned.
- We found tons of refactoring issues that are apparent once the coding is done, that we could have caught much earlier. They are now enhancement bugs in BugZilla but they could have already been fixed.
- I'm not one to make excuses, but there are two reasons for our unimpressive results concerning testing
- Testing an Eclipse plugin is hard. Running the tests in cruisecontrol took us 20+hours. Even now, our framework (took 15+ hours of tweaking for setUp() and tearDown()) seems to have issues with how elements are initialized. We try to access things that are usually available, but fail miserably.
- Our Quality Assurance Manager? did absolutely nothing in the last four months. Beforehand, he performed little or no QA tasks, the other team members had to do them for him. His efforts would have been crucial in ensuring a quality product.
--
Jason Kealey - 14 Jul 2005