AI Game Development: Teaching AI the Fundamentals of 3D Level Design
We’ve all seen the headlines.
A new AI model is released, someone writes a single prompt, and suddenly there is a video showing an AI-generated clone of Call of Duty, Starfield, or another major game.
The demonstrations look impressive.
They are often playable.
But there is an important detail that gets lost in the hype: many of these projects are essentially browser-based technical demonstrations. They prove that an AI model can generate something that resembles a game, but they are very different from the kind of large-scale 3D games that players spend dozens or hundreds of hours playing.
That distinction is exactly what interests me.
Like almost every gamer, I’ve always wanted to make my own game. Not because there is a shortage of games coming out every year. Quite the opposite. There are more games than most of us have enough time to play.
What I want is something personal.
Something with its own identity, its own atmosphere, and a distinctly Eastern European flavor.
The problem is time.
I know C#. I understand Unity reasonably well. I can work with Blender, and I can do some basic texture work in Photoshop.
But I simply don’t want to spend several years building a game manually.
Between work, hobbies, everyday responsibilities, and the usual attention problems that come with modern life, committing to a huge long-term game-development project is difficult.
So I decided to try something different:
What if I built the game almost entirely with AI?
No manually written gameplay code.
No manually modeled characters.
No hand-created textures.
No traditional production pipeline where I spend months doing repetitive technical work.
Instead, I wanted to see how far an AI-assisted game-development pipeline could actually go.
This experiment is not about creating a complete commercial game yet.
The first question is much simpler:
Can AI build a genuinely playable 3D game environment with enough technical consistency to support a serious game?
The Goal: AI Game Development Beyond Simple Prototypes
Current AI tools are already surprisingly capable when it comes to simpler game-development projects.
They can help produce:
- low-poly 3D assets;
- visual novels;
- pixel-art games;
- simple roguelikes;
- mobile games;
- prototypes;
- basic gameplay systems;
- scripts and shaders;
- concept art;
- textures;
- animations.
For indie developers, many of these workflows are already practical.
But that wasn’t the experiment I wanted to run.
I wanted to build a third-person 3D game with approximately AA-level visual ambitions, interesting gameplay systems, multiple enemy types, potentially bosses, and an atmospheric environment.
The reason is technical.
A simple game can hide many of the problems that appear immediately when you start working with a complex 3D environment.
If AI is going to become a serious tool for game development, those problems need to be solved.
Why 3D Level Design Is So Difficult for AI
There are several reasons why 3D game development is considerably harder than generating a simple 2D game.
1. 3D Space Is Not Just a Collection of Objects
In a 2D game, the AI generally works with two coordinates:
X + Y
Distances and relationships between objects are relatively straightforward.
In a 3D game, the problem becomes:
X + Y + Z + rotation + scale + collision + navigation + vertical relationships
And that’s before we consider:
- camera position;
- player height;
- NPC height;
- walkable surfaces;
- doors;
- stairs;
- terrain;
- visibility;
- occlusion;
- physics;
- navigation meshes;
- interaction zones.
The AI doesn’t simply need to know that an object exists.
It needs to understand where that object exists in relation to everything else.
2. High-Quality 3D Assets Are Much Harder to Generate
The second problem is asset complexity.
Low-poly models are forgiving.
A character with a simple body and minimal clothing can often be generated and corrected relatively quickly.
But once you start working with:
- layered clothing;
- realistic proportions;
- detailed environments;
- multiple materials;
- complex UVs;
- physically based rendering;
- realistic lighting;
the number of things that can go wrong increases dramatically.
AI-generated 3D assets are improving quickly. Platforms such as Meshy can now generate 3D assets from text and images and provide workflows aimed at game development, including integrations with Blender and Unity.
But generating an object is not the same thing as integrating that object into a coherent game world.
That distinction turned out to be extremely important.
3. The Developer Usually Has to Solve the Integration Problem
When a developer manually creates a model, they already know why the model exists.
An AI agent does not automatically have that knowledge.
It may know that an object is:
“an old refrigerator.”
But it doesn’t necessarily know:
- whether it belongs inside or outside;
- how large it should be;
- how far it should be from the player;
- whether it should block navigation;
- whether it should be interactive;
- which visual style it belongs to;
- whether the project already contains a similar object.
This means that asset generation is only half the problem.
The other half is asset management.
My AI Game Development Stack
For the experiment, I selected a relatively simple stack.
Game Engine: Unity
Unity was the obvious choice because I already had some experience with it and because it provides a mature ecosystem for third-party tools, assets, navigation, animation, physics, and AI-assisted development.
Unity also continues to expand its AI and developer tooling ecosystem, making it an interesting platform for experimenting with AI-assisted workflows.
3D Generation: Meshy
For AI-generated 3D models, I chose Meshy.
Its current workflow supports text-to-3D and image-to-3D generation, with export options and integrations aimed at production workflows.
3D Editing: Blender
Blender became the main tool for inspecting and correcting generated assets.
The important distinction is that Blender isn’t necessarily where I wanted to create everything.
It became the place where generated content could be:
- inspected;
- cleaned;
- resized;
- rigged;
- exported;
- corrected;
- prepared for Unity.
Animation: Mixamo
For character animation, I used Mixamo.
Adobe’s official documentation confirms that Mixamo can automatically rig uploaded characters and provides an animation library that can be applied to them.
This is exactly the sort of repetitive work that makes sense to delegate to an existing tool rather than rebuilding an animation pipeline from scratch.
AI Development Agent: Cursor
For the first experiments, I used Cursor as the main AI development agent.
It wasn’t necessarily the final solution.
It was simply the tool I had available and wanted to test first.
Later, I planned to compare the workflow against Codex and eventually a custom-built agent harness.
A Small Amount of Cheating
There was one important exception to the “everything generated by AI” rule.
For the early technical experiments, I used some free environment assets and existing game-development resources.
This was deliberate.
The objective wasn’t to prove that AI can generate every single polygon from scratch.
The objective was to test whether AI could assemble and manage a complex playable environment.
That distinction matters.
A game-development pipeline contains many different problems:
- asset generation;
- asset cleanup;
- animation;
- level design;
- gameplay programming;
- navigation;
- physics;
- lighting;
- interaction;
- balancing;
- optimization.
If I attempted to solve all of those simultaneously, it would become impossible to identify which component was actually failing.
So I temporarily used existing assets to isolate the harder problems.
Unity Version Compatibility Became an Immediate Problem
My first experiments with downloaded controllers and older FPS/gameplay systems immediately exposed another problem:
software compatibility.
Unity projects are highly dependent on their engine version and package ecosystem.
A controller designed around an older version can bring dependencies that are no longer appropriate for a modern project.
One of my early experiments pulled in an outdated navigation implementation.
That dependency created problems throughout the project.
Modern Unity projects use the newer AI Navigation package rather than relying on the older NavMesh workflow in the same way.
Trying to rewrite the entire controller around outdated assumptions turned out to be a bad idea.
The lesson was obvious:
AI cannot magically solve incompatible foundations.
If the underlying asset was designed for an old engine architecture, an AI agent still has to understand the difference between the old and new systems.
The Problem With Building a Level From Random Assets
My next experiment was to construct a playable scene almost entirely from ready-made assets.
Technically, there were enough assets for the chosen theme.
But visually, the result looked terrible.
Different assets had different:
- proportions;
- texture resolutions;
- material properties;
- lighting assumptions;
- polygon densities;
- artistic styles.
This becomes particularly obvious in large environments.
A single object can look excellent.
Ten objects from ten different asset packs can look completely disconnected.
Can AI Fix the Visual Style?
My first idea was simple:
Why not use an image-generation model to stylize and upscale the textures?
The results were surprisingly good at first glance.
The textures looked cleaner.
Some details became more coherent.
But there was a major problem.
PBR broke.
Physically based rendering relies on several maps that describe how a material should interact with light.
Depending on the workflow, these can include:
- Base Color;
- Normal;
- Roughness;
- Metallic;
- Ambient Occlusion;
- Height or displacement information.
If an AI model changes the visual appearance of the base texture while leaving the physically based material maps unchanged, the object can become visually inconsistent.
For example, the AI might add:
- cracks;
- scratches;
- chips;
- holes;
- surface damage.
But the normal or roughness maps don’t necessarily contain those changes.
The result can look convincing in a screenshot while behaving incorrectly under real-time lighting.
This was an important lesson:
A game texture is not just an image.
Another Problem: Some Assets Were Basically Cardboard
I also discovered something else.
Some environment assets were not actually full 3D objects.
They were extremely thin meshes with textures stretched across them.
For decorative background elements, this is perfectly acceptable.
For a playable environment, it isn’t.
A player can walk around the object.
A camera can approach it.
Lighting can expose its limitations.
Collision can reveal the problem.
And suddenly the “building” is basically a textured sheet.
This is one of the fundamental differences between a visual demonstration and an actual game environment.
AI Iterations Create Their Own Technical Debt
The longer I continued editing the environment, the more small inconsistencies appeared.
This is another familiar AI-development problem.
One missed operation during an interaction with Blender or Unity might not cause an immediate failure.
Instead, it creates a small inconsistency.
A few iterations later, that inconsistency becomes a larger problem.
Eventually you have:
- objects that don’t line up;
- missing references;
- incorrect scales;
- duplicated assets;
- broken relationships;
- inconsistent naming;
- misplaced colliders.
AI can make changes incredibly quickly.
That also means it can accumulate mistakes incredibly quickly.
The Biggest Problem: AI Doesn’t Really Understand 3D Space
My first serious experiment was intentionally simple.
I opened Cursor, gave it a detailed specification, and let it work.
The specification was created with the help of another AI research tool and described a small survival-horror prototype set in an ordinary Eastern European city.
It wasn’t my final game concept.
It was a technical test.
The scene needed:
- buildings;
- streets;
- vegetation;
- NPCs;
- enemies;
- interactive zones;
- a playable character;
- basic gameplay systems.
After a couple of hours of experimentation, Cursor had produced a working prototype.
There were enemies.
There was a player character.
There was a large environment.
There were buildings.
There were interactive areas.
And most importantly:
it was actually playable.
But then I noticed the fundamental problem.
AI Could Read the Scene — But It Didn’t Understand the Scene
The agent could inspect Unity data.
It could read object names.
It could read transforms.
It could create objects.
It could modify components.
But that doesn’t mean it understood the physical structure of the world.
It didn’t really understand what a normal city looks like.
The result was technically a city.
But the buildings were positioned in ways that no real urban environment would normally allow.
Roads didn’t always make sense.
Buildings were sometimes too close together.
Objects didn’t have consistent spatial relationships.
The AI knew the coordinates.
It didn’t understand the meaning of the coordinates.
That distinction turned out to be the central problem of the entire experiment.
Teaching the AI What a City Actually Looks Like
I decided to attack the problem from a completely different direction.
Instead of asking the AI to invent the city, I gave it real-world geographic information.
I exported detailed geographic data from OpenStreetMap through its API, including information about:
- building locations;
- building types;
- building heights;
- roads;
- waterways;
- points of interest;
- vegetation;
- local coordinates.
OpenStreetMap is particularly useful for this type of experiment because its data is structured rather than being merely a visual map.
The result was transformed into a more universal JSON-like representation.
For example:
{
"environment": {
"water_bodies": [
{
"id": "river_example",
"type": "river",
"width_avg_m": 15.0,
"coordinates_local_xy": [
[-427, 600],
[-390, 550],
[-237, 160]
]
}
]
},
"road_network": [
{
"name": "Main Street",
"class": "residential_main",
"surface": "asphalt",
"width_m": 7.0,
"points_local_xy": [
[-220, 153],
[-200, 206],
[48, 206]
]
}
],
"buildings": [
{
"id": "bld_residential_01",
"category": "residential",
"floors": 3,
"height_m": 10.5,
"roof_type": "gabled",
"material": "brick",
"footprint_local_xy": [
[635, 132],
[643, 125],
[653, 136]
]
}
]
}
Now the AI wasn’t simply being told:
“Build a city.”
It was given structured information about how the city was physically organized.
That made a huge difference.
Real Cities Are Not Automatically Good Game Levels
There was another problem.
A real city is not necessarily a good game level.
If you reproduce a real city at realistic scale, the player may have to walk for several minutes between locations.
That’s perfectly normal in real life.
It’s often terrible level design.
Game environments need to balance:
- realism;
- navigation;
- pacing;
- encounter density;
- exploration;
- landmarks;
- player orientation;
- traversal time.
A real city may contain thousands of buildings.
A game level might need only a few dozen.
So the next challenge became:
Teach the AI how to simplify reality without destroying its logic.
The AI needed to understand the structure of a city while also understanding that a game level is a compressed representation of reality.
Iteration Two: A Smaller Controlled Environment
For the second iteration, I deliberately reduced the scope.
The scene contained approximately:
- 10–15 buildings;
- several abandoned vehicles;
- one interior room;
- a simple interior layout;
- additional props;
- several interactive elements.
The problem returned almost immediately.
Cursor still had difficulty reasoning about three-dimensional placement.
Buildings intersected.
Cars were sometimes tiny.
Other cars were enormous.
Some vehicles ended up blocking doors.
An object that looked fine numerically could be completely absurd visually.
This confirmed something important:
The problem wasn’t simply the choice of AI model.
The problem was the representation of spatial information.
Teaching the Agent to Calculate Spatial Relationships
The solution was surprisingly simple.
I asked Cursor to create a C# utility that calculated object placement relative to neighboring objects.
Instead of letting the agent place every object based purely on textual instructions, the project gained a small layer of deterministic spatial logic.
The AI could now reason in terms of:
- object position;
- object dimensions;
- neighboring objects;
- minimum distances;
- player scale;
- zone boundaries.
In other words, I stopped asking the LLM to be a physics engine.
I gave it tools that could perform the calculations reliably.
This is a critical principle for AI game development:
Don’t force an LLM to perform deterministic calculations that software can perform more reliably.
The LLM should decide what needs to happen.
The game engine or utility code should calculate how it physically happens.
The Asset Knowledge Base
The second solution was even more important.
I created a classification file describing the available assets.
For example:
| Asset | Meaning | Usage | Status |
|---|---|---|---|
kiosk |
Street kiosk | Outdoor streets only | Used |
low-poly-doors |
Doors | Building entrances | Staging |
military-box |
Ammo crate | Interior only | Used |
panelka |
Soviet-era apartment silhouette | Exterior / skyline | Staging |
old-fridge |
Refrigerator | Interior only | Staging |
old-tv |
CRT television | Interior only | Used |
This looks simple.
But it fundamentally changed the AI’s capabilities.
The agent now had a small knowledge base for the game world.
It could determine:
- what an asset was;
- where it could be used;
- where it should not be used;
- whether it was already in the project;
- whether it was currently being used;
- whether it belonged to an interior or exterior;
- whether it was heavy and should be placed sparingly.
This is much more useful than simply dumping thousands of asset files into an AI context.
From AI Generation to AI World Management
This was the point where the experiment started becoming much more interesting.
The AI was no longer just generating things.
It was beginning to manage a structured game world.
I also added an agent responsible for tracking:
- what was placed;
- where it was placed;
- how many copies existed;
- which assets were being overused;
- which zones were becoming too crowded.
This solved another classic problem with procedural and AI-generated environments:
repetition.
If an AI has a collection of 20 props and no rules, it will happily use the same three objects everywhere.
The result quickly looks artificial.
A knowledge base and usage tracker provide constraints.
And constraints are extremely important in generative systems.
What AI Managed to Build Immediately
The results were actually better than I expected.
Cursor quickly connected the Unity character controller with enemy animations.
The first iteration already contained functional NPCs.
The agent could also use the specification to create:
- interactive zones;
- dialogue NPCs;
- enemy encounters;
- restricted pursuit areas;
- randomized enemy spawning;
- placeholder cinematic triggers.
The survival mechanics were another interesting experiment.
Because the original specification explicitly described a survival game, the agent implemented systems involving:
- fatigue;
- hunger;
- thirst;
- player actions;
- survival-state changes.
The important point isn’t that an AI can write these systems.
We already know that.
The interesting part is that the AI could integrate them into the prototype based on a sufficiently detailed design specification.
The Three Biggest Lessons From AI Game Development
After these experiments, three lessons became clear.
Lesson 1: AI Needs a Structured World Model
The most important thing wasn’t the model.
It was the documentation.
An AI game-development agent needs to know:
- what objects exist;
- what those objects mean;
- how large they are;
- where they can be placed;
- what they can interact with;
- what zones they belong to;
- how they relate to neighboring objects.
Without this information, the AI is essentially improvising.
And improvisation is dangerous when you’re building a 3D environment.
Lesson 2: Creating Is Easier Than Fixing
This is probably the most universal lesson.
Building a clean pipeline for generating a new asset is often easier than repairing an asset produced by someone else.
The same applies to code.
The same applies to levels.
The same applies to AI-generated projects.
If the pipeline is designed correctly from the beginning, everything downstream becomes easier.
If you start with incompatible assets, outdated packages, inconsistent naming, incorrect scales, and missing documentation, the AI spends most of its time repairing problems rather than creating new things.
Lesson 3: Your Idea Must Be Explicit
The third lesson is perhaps the most important.
Don’t expect the AI to understand what you mean.
Write it down.
If an object has a specific purpose, document it.
If a building has a specific scale, document it.
If an area should contain only certain types of props, document it.
If a mechanic has specific rules, document them.
If the player should never enter a particular zone, document it.
The AI doesn’t need to guess.
The better your game design documentation, the more useful your AI becomes.
AI Game Development Is Becoming an Engineering Problem
The biggest misconception about AI game development is that the model itself is the entire solution.
It isn’t.
The model is only one component.
A serious AI-assisted game-development pipeline looks more like this:
Game Engine
↓
Project Documentation
↓
Asset Knowledge Base
↓
AI Agent
↓
Deterministic Tools
↓
3D Asset Generation
↓
Validation
↓
Automated Testing
↓
Human Review
This architecture is much more promising than simply giving an LLM a giant prompt and asking it to “make a game.”
The Future of AI Level Design
I think the next major development in AI game development will not simply be better text-to-game generation.
It will be better world representation.
AI needs more than descriptions.
It needs structured information about:
- geometry;
- scale;
- navigation;
- physics;
- gameplay zones;
- object relationships;
- player movement;
- environmental rules.
Once an agent can combine semantic knowledge with deterministic spatial systems, much more sophisticated game-development workflows become possible.
And this is where tools such as MCP become particularly interesting.
Instead of forcing the LLM to understand an entire application through raw text, MCP-style integrations can expose specific tools and structured information to the model.
That approach is already becoming common in AI development environments. For example, Zed currently supports MCP servers, external agents, configurable tools, skills, and project instructions, allowing the developer to control exactly what the agent can access.
The same philosophy can be applied to game development.
The AI Shouldn’t Control Everything
Ironically, the best AI game-development pipeline may be one where the AI doesn’t have complete freedom.
Give it too much freedom and it improvises.
Give it too little and it becomes useless.
The sweet spot is a controlled environment.
For example:
AI decides:
- which asset to use;
- where an encounter should occur;
- which NPC should spawn;
- which gameplay system needs modification;
- what kind of environment should be created.
Deterministic tools decide:
- exact coordinates;
- collision;
- scale;
- pathfinding;
- physics;
- valid placement;
- asset compatibility.
Human decides:
- whether the game is actually fun;
- whether the level feels good;
- whether the atmosphere works;
- whether the mechanics are interesting;
- whether the final result is worth shipping.
That division of responsibilities makes much more sense to me than expecting an LLM to do everything.
Can AI Really Build a Complete Game?
After this experiment, my answer is:
AI can already build surprisingly large portions of a game.
But that doesn’t mean you can give one prompt to an AI model and receive a polished AAA game.
The difficult part isn’t necessarily generating code.
It isn’t even generating assets.
The difficult part is maintaining consistency across thousands of interconnected decisions.
A real game contains:
- systems;
- levels;
- assets;
- animation;
- physics;
- UI;
- audio;
- narrative;
- progression;
- balancing;
- optimization;
- testing;
- technical infrastructure.
Every component affects the others.
That is why game development is fundamentally different from generating a standalone demo.
The Real AI GameDev Challenge
The future isn’t necessarily:
“AI generates an entire game from one prompt.”
It may instead be:
“AI becomes the technical team that operates inside a structured game-development pipeline.”
That is much more realistic.
One AI agent handles code.
Another manages assets.
Another checks level geometry.
Another validates navigation.
Another tracks performance.
Another maintains documentation.
The developer remains the director.
This is much closer to how I now imagine AI-assisted game development evolving.
Final Thoughts
The most interesting result of this experiment wasn’t that AI could create a playable Unity scene.
We already know that AI can generate impressive demonstrations.
The interesting part was discovering where AI starts to fail.
It struggles when the world is poorly represented.
It struggles when spatial relationships aren’t explicit.
It struggles when assets don’t have metadata.
It struggles when project documentation is missing.
It struggles when deterministic calculations are left entirely to probabilistic reasoning.
But once those problems are addressed, the results improve dramatically.
That leads me to a simple conclusion:
The future of AI game development isn’t just about smarter models.
It’s about giving those models better representations of the world.
A good AI game-development system will need:
- a structured world model;
- an asset knowledge base;
- deterministic spatial tools;
- a well-defined Unity project;
- automated validation;
- persistent documentation;
- a capable AI agent;
- and a human who actually understands game design.
The AI can generate the pieces.
It can connect them.
It can debug them.
It can iterate.
But someone still has to teach it what the world is supposed to mean.
And that, surprisingly, may be the real beginning of AI-powered level design.