Testing/Play-Testing IGN™ Materials

Testing: Material testers use Google Groups to discuss Incarna game issues. This is currently publicly viewable. A set of Play Test resources are available online.

Play Test Model (2022-2023)

Incarna’s first play testing was done in 2002. All testing groups have their own gaming contract.

Participation Basics: Access to pre-production and production (design, development, testing/play-testing, and publishing) elements (content, materials, processes, etc.) are all bound by the standard NDA terms.

Coordinator: This is a game designer, project manager or designated representative of the Incarna Gaming Network. They will manage communication
and other specific duties as dictated by their responsibilities between the group and Incarna interests.

  • Tester: These are designated and assigned people who will be testing rules and materials as assigned by the Play Test Coordinator.
  • Initial Design

    The final responsibility for good game play lies in the hands of the game producer.
    Prior to any code or materials being written, the designer must look at the proposed game through
    the filters of " game play" to assess its playability. This can be difficult if the
    game designer, developers and producers are not the same party and do not have adequate measures
    to enable a dynamic, fast response development process. If the initial game materials arrive
    to play testers in a hap-hazard way or format, without any thought to playability prior to
    initial testing, the design will be flawed and doomed from the start. Incarna went through a
    year long concept phase wherein players and designers discovered that was important to players
    and what approach designers should take to accommodate the priorities set by players. The proper
    approach would allow the game a level of flexibility/adaptability for easy alteration.

    Starting and Controlling the Test Process

    Play testing will be done in two different ways: Observe players while using the materials (in-house),
    and granting access to the game and elicit feedback via questionnaires and interactive online elements
    (email, chat, etc.).

    In house testing is very time consuming, and requires a lot of formal and involved direct observation
    of players using the materials and playing the game of a prolonged period of time. The focus of this is
    playability, and not market or demographic impact. The recruitment of in-house play testers in usually
    accomplished by canvassing friends and known elements. This usually allows a reasonable degree of cooperation and
    lack of tension as the participants are known and social interplay is reduced to elements which can be filtered out during test
    result review. Using community (game community) resources (computer shops, game shops, college campus) such as news boards
    can also be an effective means of finding play testers with like minded interests.

    All potential testers should be questioned about past experience and games they like to play. People may enjoy
    RPG games, but only within a strict set of guidelines such as a particular genre (sports, fantasy
    or sci-fi perspective) or game technology or setting and may not be good candidates. While playability
    does not focus on demographics, initial play testers should be pulled from the your most common demographic
    element. Also, play testers must understand that in the case of testing, character development and the value of
    their actual role playing skills is reduced, while the emphasis is stronger on technical knowledge, timeliness,
    and effective communication. All play testers should be made aware of the commitment, preferably in a play testing contract,
    prior to their acceptance of an offer.

    Review Timing

    A game needs to be stable enough that the play tester doesn’t spend too much time noting bugs in the mechanics,
    yet not developed to the point that changes cannot be made without affecting the entire underpinnings of the game.
    A minimum of 3 months time should be allotted for each play testing cycle, as the availability of play testers
    may vary. As much feedback mechanisms, and online surveys and questionnaires should be made available so that in-house play testing
    can focus on important aspects during the time the group is together. Having more than enough people for a single play testing
    group is usually a positive factor, as scheduling that many people can be difficult.

    Testing Oversight

    In-house testing should always have a a high ratio of developers and designers to testers.
    These staff should be available to answer questions, observe, and note issues that come up in play for later review.

    • What materials cause the most confusion? Where do the testers ask for the most assistance?

      This may vary by tester, if one person continually does not understand a concept or know how or where to find help, it should
      be noted this is not likely to be a problem with the game materials but lack of adequate skills of the tester.
    • What kinds of features do the play testers have the most questions about?

      Identify key categories of questions and ask if the layout, presentation, organization or complexity can be changed.
    • Do play testers get frustrated with the game easily? How closely does their frustration level relate to their skill level?

      Benchmarks need to be established prior to bringing in the play testers; additional benchmarks will be added to as testing proceeds.
      If a game scenario uses puzzles which directly relate to game facets, establish a minimum and a maximum amount of time for the play testers to solve each puzzle.
      Timing tester answers and ability to find information needed for the game can be an important tool for evaluation.
    • Do play testers like the game?

      On test is their ability to cite specific pieces of information about a setting or rules in the game that they liked.
      Requests for more materials will give you an idea of a measure of success. If there is an aspect of the game that is near universally
      disliked, a Quality Control meeting needs to be held and the assessment and impact of changing the aspect needs to be
      measured against the usability test results.
    • If the play testers echo sentiments made during the earlier testing phase, and the items criticized were not fixed or changed, not enough attention has been paid to the testers.
      Again, a Quality Control meeting should be arranged to discover where and why the process has broken down or the materials have not been changed.

    Testing Goals

    • Play testing should provide the materials producer/owner with as much information as possible for making the necessary game play changes.
      The designers and producers need to hear overall opinions which may shape the action or playability of the game.
    • The play testing coordinator needs to make sure that the groups have adequate time to review all materials assigned.
      Oversight and communication between test groups and producers/designers are the primary concern.
      To this end, they must make sure all the right tools are in place and working properly. The end goal for them
      is to establish new and refine exiting processes to make sure this happens.

    Standard Evaluation Guidelines

    Testing Results

    • Game tweaks and changes. Information needed for the game to enhance or maintain its viability in the market.
    • White Papers: Out of play testing should come several position papers (called "white" papers) which clearly spell out decisions
      made to persist game aspects, change game aspects and explain or justify the underpinnings of the game mechanics.

    Guiding Principles

    Although the general flavor of these principles is from the Play Test Coordinator perspective,
    they are good general guidelines for testers to adhere to when dealing with other members of their own group.

    Listen

    Do not lead testers to solutions or opinions that they have not professed. Listen to what is being said and review
    with other designers or producers. It is important to try and solicit answers to problems, like "What would you do instead?",
    but do not provoke leading questions or answers until you have had a chance to discuss with others.

    Do Not Be Defensive About Criticism

    While there may always be those testers liking nothing and advocating a re-vamp of the entire game,
    most play testers will have constructive feedback. Some of this may be negative. Not all of it
    will have easy solutions. Do not mistake frustration with personal attack – it’s easy to do if a tester
    has problems and their patience runs low as a result of lack of response or general disaffection with
    an aspect of testing or the game itself.

    Give Clear Instructions

    Make sure all instructions you give the play testers are clear. If possible, have documentation to distribute
    or some other from of persistent (web pages, emails) information which can be referenced by the tester to
    understand what is expected of them and the scope or goal of their involvement.

    Do Not Waiver

    In the face of the bombardment from play testing resources, do not change positions or play soft to multiple parties.
    When expressing opinions or rulings, stand firm on positions and be able to back up the official position
    with a knowledge of the materials and any documentation about them.

    Do Not Perpetuate Legacy Problems

    Just because it’s difficult or a lot of work, do not discount feedback for testers on any aspect.
    If past responses to change requests have been that it would take too much time, change the entire game engine,
    or cause a lot of problems, that is no reason to not accept and review them as equally as any other.
    A lot changes in technology and ideas from one year to the next, and a refactoring or engineering may not take as much
    as previously thought.

    Reward Testers

    Testers provide the feedback for the product to remain viable in the market place.
    Some from of reward is always necessitated. The exact from will depend on the closeness
    of the group, the financial situation at the time and the specific desires of the individual play testers.
    Offering reward options is usually better than granting everyone the same reward.

    Teamwork

    If every piece of the play test process is not running well, the results and responses
    to them will be flawed. Much of the entire process boils down to all involved working well together.

    Play Test Coordinator Responsibilities

    Assign product and specific rule reviews. Manage play test group assignments.
    Reporting summarized results back to game designers in a clear and concise manner.
    The coordinator must consistently send out
    notice regarding where and when the next session will be for each group. This may include what will be tested but
    will certainly include what you will need to prepare beforehand and bring to the
    session. This notice may state specific instructions (e.g. Play Tester A prepare a magus with Occult Aptitude 7
    and SCH 55), freeform instructions (e.g. make a fantasy based character), or
    something in between. Play testers should follow the instructions provided so that when the time comes
    the session may begin without undo delay.

    The coordinator will create and supply materials for feedback.
    These materials will be made available online in the play tester section
    of the Incarna web site or some other shared venue.

    Tester Credit