Replying to DoW1 Mod Project Testing Scenarios Checklist

If you run the game from STEAM, right-click [Properties] on the game, and then click on [SET LAUNCH OPTIONS...], and put the -dev there.

 

As for the DVD version, you must go into the shortcut that run the game (its properties), and add the -dev there.

"Firstly. ensure you run your game with a -dev command as follows:

start soulstorm.exe -dev -nomovies -modname [your mod name]

Generally I create a batch command file (.cmd) in the location where your Soulstorm is installed to ensure it runs only the mod I am testing. I also add in some logic to remove useless debug files as well. Note this example of a batch command file:"

Where do I find this -dev command?

- Intermediate and Advanced AI issues really are not "in-scope" for this one as it "presupposes" the mod team did all the vetting long time ago for CTDs because it does not take long to check for such major AI issues before release. If the testers notices a CTD that would be assumed the devs be notified but CTDs are not necessarily related to AI alone.

 

- Yes.. Missing Sounds are mentioned in my dox.

 

- Hmm.. ok I'll make a mention of how to bring the console down BUT remember, all console messages are still logged to warnings.txt so that part is already covered so long as the tester submits that file to the devs..

 

- Balance testing.. maybe.. might be, again, out-of-scope for this doc: this is meant for more deeper problems.

 

Maybe its just simpler to ask for the entire last \logfiles\[tested instance] and they'll be no need for testers to record only specific errors and send them to the devs: just zip up the logs and the devs will examine it themselves.

I am not sure if you want replies here, I can delete this later. I would suggest you are missing something like an 8 way mod race Vs mod race on high speed to check for AI issues. These are normally indicated by a ctd.

If you hear an ear splitting beep then that is indicative of a missing sound file.

A fatal scar error will have an error message. You can either screenshot this by bringing down the console (ctrl+shift+@) or get the message from warnings log (you cannot do this whilst the game is running, the logs only complete on game close).

Also this would not be a good guide for balance testing.

Revision 0.5
 
This brief instruction list is meant to better test most facets of a DoW1 mod project specially once which is a faction mod OR one that has many moving parts which involve 3d models.
 
What we want to achieve is to get as much info for the mod in question so we ensure quality and cover as many scenarios as possible.
 
 
Testing Setup
 
Firstly. ensure you run your game with a -dev command as follows:
 
start soulstorm.exe -dev -nomovies -modname [your mod name]
 
Generally I create a batch command file (.cmd) in the location where your Soulstorm is installed to ensure it runs only the mod I am testing. I also add in some logic to remove useless debug files as well. Note this example of a batch command file:

del *_ErrorLog.txt
del *_MiniDump.dmp
del %~dp0mymod\data\ai\mapdb\*.dat
start soulstorm.exe -dev -nomovies -modname MYMOD

In the above, we're removing any useless error and dump files in the root of your Soulstorm folder + extra AI mapdb info (if you had that game option enabled but its generally best to clear those dat files out).
 
 
So where are these Debug Logs?
 
What adding -dev does to the command line is it opens up a folder called \logfiles in the root of where your Soulstorm is installed.
 
In this logfile you will find some useful debugging text files that will help you determine various issues in the mod you are playing so that the devs can more easily and effectively correct them.
 
The ones you will need to be most aware of are as follows:
 
ailog.txt -- dumps AI specific messages found during your tests like invalid weapon hardpoints assigned to the AI
artmessages.txt -- dumps art-related issues warning of but not exclusively only about missing texture, FX, events, or invalid or bad actions being called by assets in the project. We will be using this one the most.
designmessages.txt -- dumps AE or Attribute Editor errors where the core logic has some issues (ie. duplicate entries or invalid modifiers). Generally this file should not appear in the folder but if it does LET THE DEV KNOW!
scarlog.txt -- dumps Scar-only items so can be useful when seeing or troubleshooting specific scar logic (ie. HEROES errors will show here if it spots an error when the HEROES game mode is active).
soundmessages.txt -- dumps sound errors and warnings pertaining mostly to missing sound files required or called by a specific asset.
warnings.txt -- dumps generic high-level debugs showing the step-by-step load cycle of the game: among the key items here are revealing missing badges, banners, 3d models and textures, and general console messages.
 
The warnings.txt should always be looked at first noting anything unusual especially just before the mod being tested loads into the map. Generally, nothing else useful will show after the map is loaded as most the during-the-game dump is not entirely useful (unless your game CTD then, sometimes, something may show there at the very end but not always).
 
Hint: to view real-time warnings.txt dumps as you are playing the game simply hold the following keys down together (shift+` or ctrl+shift+@) and the main console will be shown.
 
Remember: only once you finally did all your testing in-game that the above debug logs will be unlocked (they are locked during the game) so once you exit the game can you then begin to read them.
 
 
Testing Process and Methodology
 
Generally, I would recommend loading a map like 2p_Velvet_Duress since it has most of the elements we need to test with although 2p_Fallen City has a Slag Depot to test if your faction mod has a Large Generator but generally I use 2p_Velvet_Duress. 
 
Also, I always play against an Ork AI (disabled so it will never attack) and, most times, I will make its Ork HQ invulnerable so I can run many tests against it without actually taking down their HQ accidentally.
 
Further, I run with an option to continuously give myself both requisition and power so my testing is never interrupted or slowed due to economic reasons + I run the game @ setsimrate(100) which is extremely fast (not recommended for slower machines) as well as an option that everything builds instantly (except upgrading hardpoints). We just want to test EVERYTHING we can think of quickly so we're not bogged down.
 
Notation: I can supply these "tools" to allow for the above features to be unlocked if you request.
 
Now, when testing, if what you are testing has many moving parts and/or branches/tangents then you want to ensure you test one part of it to the end of its logical sequence until you cannot test anymore THEN restart the game and run the other branches/tangents: this way you will get a broad test result which will greatly help the devs fix all outstanding and, perhaps, minute issues not discovered by regular means.
 
Remember, with all of the above set, the game is RECORDING YOU so anything you do will be seen by the game's back end and dumped to those debug files.
 
 
What are we looking for then?
 
The following is a general list of what to test FOR EACH UNIT in the game. Buildings generally do not apply here apart from odd icon placement, pink icons, or unusual UCS info used.
 
Heroes Support
Run with HEROES Game Mod and simply check each unit has HEROES showing on the right side for that unit. This will ensure HEROES is being used for all units and none were accidentally omitted.
 
Cover Icons
Play on maps with light, heavy, or negative cover and ensure each unit has the cover icon showing correct "on the top" of the unit.
 
Treads moving on vehicles that have them
Vehicles like Rhinos and Predators use animated tracks so ensure they move/animate when the vehicle moves.
 
Ranged Attack anims and FX
Put a fully loaded/upgraded squad in front of an enemy building ie. as mentioned, an Ork HQ is the best, and let the squad fire at range all weapons - not any missing any FX or weirdness in the firing process.
Note: anything missing in terms of FX logic, FX textures, or events will show up in artmessages.txt so please note this to the developer.
 
Quick examples found in the artmessages.txt of what we are looking to correct are:

FX -- Texture 'DATA:ART/FX/INQUISITION_DAEMONHUNT/TEXTURES/LIGHT_GLOW_BLUE' is missing, using default!

FX -- type (inquisition_daemonhunt\units\grey_knights\blue_light) not found. Replacing it with default FX (badfx).
    blue_halo_psycanon_l
    art/ebps/races/mymod/troops/my_new_3d_model

EVENTS -- 'Weapon Damage': Unable to load event 'art/events/guard/abilities/fanaticism'!
EVENTS -- 'Combat Weapon': Unable to load event 'art/events/missile pods'!
EVENTS -- 'Event Manager': Unable to load event 'art/events/structure_fx/relocation'!
EVENTS -- 'Deep Strike': Unable to load event 'art/events/1'!

'ranger': XRef'ed Anim 'vis_no_flag' in model 'art/ebps/races/mymod/troops/my_new_3d_model' is missing!

RENDER ANIM -- Art/EBPs/Races/mymod/Troops/my_new_3d_model: Unable to open file!

In the soundmessages.txt a quick example of what we want to fix is:

Failed to preload patch 'races\inquisition_daemonhunt\weapons\nemesis_force_weapon'
    sound_nemesis_attack
    art/ebps/races/mymod/troops/my_new_3d_model

'DATA:Sound\Speech\Races\mymod\my_new_unit\strategic_point_capture\401006.fda': 'Not a valid chunky file!'

Warning mixing empty blocks for DATA:Sound\Music\MUSIC_MYMOD_THEME.FDA. Possibly error in streaming.

Note: if you hear a beep during your game testing, that usually means a sound file is missing which was being called by the project somewhere: the above is an example of one such sound error being recorded and dumped to the soundmessages.txt.
 
In the warnings.txt a quick example of what we want to fix is:

LOCALIZER -- Requested string ID '9000500' does not fit in a valid key range!
LOCALIZER -- Requested string ID '9000000' does not fit in a valid key range!
LOCALIZER -- Requested string ID '9000501' does not fit in a valid key range!
LOCALIZER -- Requested string ID '9000502' does not fit in a valid key range!

RENDER ANIM -- Art/EBPs/Races/mymod/Troops/my_new_3d_model: Unable to open file!

RENDER ANIM -- 'art/ebps/races/mymod/texture_share/my_new_3d_model_texture': Unable to open file!

 
Melee Attack anims and FX
Put same squad as above into melee against an enemy building ie. again, using the Ork HQ and look for any model glitching especially looking for invalid special_attack problems when engaging in direct melee.
Note: same applies as above and note anything in the artmessages.txt to the developer. 
 
Again, quick examples found in the artmessages.txt of what we are looking to correct related to close combat are:

'art/ebps/races/mymod/troops/my_new_3d_model': Missing game action 'special_attack_3'!

Use all abilities on Units
Try to use all abilities available to all units in the mod: this will allow us to ensure all abilities anims and FX are properly working with the assets involved.
 
A quick example found in the artmessages.txt of what we are looking to correct related to abilities would be:

'art/ebps/races/mymod/troops/my_new_3d_model': Missing game action 'battlecry'!

Jump, Teleport, and Capture.
Make sure you attach a Commander unit like Skull Probe or Apothecary to a squad that supports those "features" and note what happens to attached Commander.
 
A quick example found in the artmessages.txt of what we are looking to correct related to both jump/teleport and capture would be:

'art/ebps/races/mymod/troops/my_new_3d_model': Missing game action 'jump_setup'!)
'art/ebps/races/mymod/troops/my_new_3d_model': Missing game action 'capture_strategic_point'!)

Note: even if the model does not glitch or seem broken when starting to Jump, Teleport, or Capture note how it looks as it happens. Does it look correct? ie. generally, when teleporting or capturing a point a unit should kneel with hands over its face preparing to teleport or as it is capturing.
 
Also note, does the unit or squad itself support capturing point? Does it plant a flag at the point or glitch out? Again, this will likely be recorded in the logs BUT sometimes a model still has a proper capturing animation (planting the flag) or capping animation (as point is being slowly captured) but it is not animated correctly or just looks.. odd. This can be subjective of course.
 
Pink Icons
Check all upgradable builders, squads, commanders, and vehicles that can upgrade hardpoints and ensure they do not ever show any PINK icons for the unit itself or weapons.
 
Minor UCS checking
Note all info for everything and ensure its both accurate and not too cluttered. ie. If a unit has a limit of 1 this should generally be mentioned in the info right at the last line. The second last line should basically read what this unit is effective against. Again, this is up to the mod team to standardize on so could be considered, subjective.
 
Projectile Origins
For units that use projectiles like Missile Launchers take a quick note to see from what point does the projectile originate from the model? Does it appear to come from anywhere other than the actual barrel of the Launcher? Generally, its best for the projectile to originate from the muzzle center of the launcher and not anywhere else. Let the dev know if it seems off BUT please remember it is difficult to center it perfectly so only mention if it looks completely off.
 

Pintle Turrets and Side-Sponsors pivoting/rotation - Updated this section June 11, 2018

For vehicles that contain pintle-mounted weapons (ie Storm Bolters or Multi Meltas manned by a Marine) check to ensure the turret rotates correctly when firing on a target. General rule: if the top of the vehicle has a Pintle-Mounted Weapon AND it has no visible obstructions (ie. equipment or decoration) then then pintle should be able to rotate a full 360 degree firing arc. If not, then only at 180 degrees or less as WE DO NOT want the Pintle firing through anything obstructing it on the vehicle.

 

Same goes for side-mounted Sponsor weapons (ie. SM Predator's Bolter and Lascannons) where the rotation should be a max 180 degrees so when the vehicle turns away from the enemy, that side of the weapon sponsor will stop and NOT show it firing through its own hull trying to target the enemy.

 

Also, three other things about side sponsor weapons (some actually apply to top-mounted Pintle turrets also):

 

1) Ensure they swivel to face enemy fairly quickly.. I've come across some side sponsors that took FOREVER to turn so I fixed it. LR Crusaders' side Hurricane Bolter arrays were notorious for this!

 

2) Make sure the weapon fires at target WHILE IDLE! I've come across incidents where the vehicle needs to be moving to fire at target but when actually facing target AND stationary it will fire once then do nothing. Unless weapon needs to be stationary in order to fire, it must always fire at target if moving or stationary.

 

3) When both side sponsors do not fire in front of vehicle straight on but need to swivel to attack is also a no-no.. its like asking the Land Raider's two side lascannons NOT to fire when the whole Land Raider is directly facing an enemy. BOTH side sponsors should fire straight to target UNLESS there is a logical obstruction on the vehicle to block it.

 

Preform minor balance testing
While its not entirely the duty of the tester to fully vet the project's balance against other factions (vanilla or mod) try to fire weapons at enemies to see how well they do against them? If you feel they are too strong or too weak this is a nice bonus for the developers to know so they can be corrected in tandem with other test fixes going on.
 
-------
 
Essentially, play every aspect of the mod and its different branches/tangents using the testing methods above then, once completed, zip up the \logfiles\[instance-recorded session] that was captured and send it to the devs of the project. This will ensure all aspects of the project are covered.
 
I'll continually update the above document so expect more revisions.