Perforce integration in a AAA game engine

Perforce integration in a AAA game engine
The Strength of the C++ API
Marc Desfossés
Lead Tools Programmer
Jean-Sébastien Pelletier
Technical Lead
Perforce integration in a AAA game engine
Ubisoft Entertainment
1

Introduction
The organized chaos of the game industry
2

History – Prince of Persia
• How it worked on Prince of Persia (2003)
• In-house source control
• What we loved
• Bigfile: File system stored in a single file
• Integrated in editor
• What did not work
• Data corruption
• Not scalable : No submit queue
3

History – Starting from scratch
Foundations of a new engine (2004)
• Full Perforce Integration
• Keep Bigfile’s concepts
• AAA game title
• Big teams
• Lots of assets
4

Game Production Workflow
Creating a AAA game
5

Engine overview
Anvil (Editor)
3DStudio Max
MoAon Builder
Photoshop
Scimitar (Engine)
Guildlib (Data Exchange)
P4
Bigfile
6

Objectives
• Performance
• Stability
• Get rid of data corruption
• Scalability
• User Experience
7

Objectives - Scalability
• Over 2 millions of files on AC3
• ~ 500 users submitting code or data
• 150 Gigs of data
• Multi-Studio support (5 studios)
8

Objectives - Scalability
9

Objectives – User experience
• Simple User Interface
• Exposing only the basics features
• Add, Edit, Delete
• Limited branching and resolve for power users
11

User Experience - Custom dialogs
Anvil Editor
12

User Experience - Custom dialogs
Visual Diff
Submit Assistant
13

Architecture
Leveraging the C++ API
14

Architecture – The Bigfile
• File system embedded in ONE file similar to a zip file.
• Database driven
• Transactions
• It contains relationships between all game objects
• Source control metadata for each file stored in FAT
• Revision number
• File status
15

Architecture – Clientview Sync Process
- Mirror bigfile and its associated perforce client
- Run commands such as fstat, flush, edit, revert, delete,
add.
- Once this is completed we can run any Perforce
command.
16

Architecture – Splitting game object files
BIGFILE
/foo/texture.psd
Perforce
Meta Data
Name = texture.psd
Object Type = Texture
Folder = foo
Data
17

Architecture – Splitting game object files
• Avoid branching when moving or renaming
• Move or Rename becomes edit of the meda-data file
• Save disk space on server
• No data file submit for rename or move
• Allow custom file statuses
• Only the meta-data file is synced
18

Architecture – File Mapping
BIGFILE
Perforce Depot
/foo/texture.psd
File texture.psd with ID 0x300012ce1
//ac3/main/d/3/00/01/300012ce1.d
//ac3/main/a/3/00/01/300012ce1.a
19

Architecture – The C++ API
• FileSys class
• Used to redirect I/O
• Lets us improve performance
• Callback call for each file when running
commands
20

Architecture – Sync Example
• P4 sync //ac3/main/…
• Callbacks in FileSys class for each file synced
• Open()
• Stat() – Return false (no temp file)
• Write() – Append buffer in memory
• Close() – Commit buffer and update metadata in
bigfile
21

Architecture – Sync Performance
22

Architecture – Custom File Statuses
• Ghosted files
23

Architecture – Custom File Statuses
• Hijacked files
• Clobbered files that can be reverted
• Skipped during sync
• Contained in a special changelist
24

Demo
25

Benefits vs Costs
Is it really worth it ?
26

Benefits
• Integrated User Experience
• Embedded Perforce operations
• Submit Assistant dialog
• 5 minutes setup
• Custom Statuses
• Ghosted files
• Hijacked files
• Performance
27

Costs
• High maintenance
• Always behind Perforce’s new features
• Shelve and Streams for instance
• Testing is complex
• Upgrades need to be fully tested
28

Final words
29

Final words
• The Perforce C++ API is powerful and lets you
manage your own File System.
• Perforce can support the load required by the
gaming industry.
• A tight integration allowed us to simplify our
users workflows.
30

Questions ?
31